{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction to Theano\n",
    "\n",
    "What is Theano? Theano is a method of describing mathematical operations that can be carried out either in Python or by low-level C code on a Graphics Processing Unit (GPU).\n",
    "\n",
    "## Simple Example\n",
    "\n",
    "To get started, let's explore a very simple example. Consider the function:\n",
    "\n",
    "$ f(x, y) = x + y $\n",
    "\n",
    "In mathematics, we write the function $f$ that takes two parameters, $x$ and $y$. The body of the function is simply $x + y$.\n",
    "\n",
    "In Python, we would write that as:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def f(x, y):\n",
    "    return x + y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "11"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "f(5, 6)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In Theano, we use the Python language as a method of writing symbolic expressions. \n",
    "\n",
    "First, we import the needed components `T` and `function`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import theano.tensor as T\n",
    "from theano import function"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we define two symbols, x and y. These are both Python variables and Theano symbols. The quoted 'x' is the Theano symbol, which is assigned to a Python variable of the same name (to make it easier to understand and help during debugging)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "x = T.scalar('x')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The term scalar simply means that it has a single value. In Theano, this is sometimes called a zero-dimensional array."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We do the same with y:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 208,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "y = T.scalar('y')\n",
    "\n",
    "# or both together:\n",
    "x, y = T.scalars('x', 'y')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, the Python variable x is of type TensorVariable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 209,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "theano.tensor.var.TensorVariable"
      ]
     },
     "execution_count": 209,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we define a Theano function. We do this in two steps. First, we describe the body of the function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 210,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "func = x + y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This looks like we are adding x and y together. However, we note that x and y are symbols... they do not yet have values.\n",
    "\n",
    "We can see that func is something very strange:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 211,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Elemwise{add,no_inplace}.0"
      ]
     },
     "execution_count": 211,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "func"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The second step in creating a function is to make a Python function. To do this, we use the Theano `function` function. It takes a list of parameters as symbols, followed by the Theano function body:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 212,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "pyfunc = function(inputs=[x, y], outputs=func)\n",
    "# or:\n",
    "# pyfunc = function([x, y], func)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 213,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<theano.compile.function_module.Function at 0x7fe7e61e0a20>"
      ]
     },
     "execution_count": 213,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pyfunc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 214,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(11.0)"
      ]
     },
     "execution_count": 214,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pyfunc(5, 6)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can take a Theano function body and show it in symbolic form, perhaps close to the original Python source:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 215,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'(x + y)'"
      ]
     },
     "execution_count": 215,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from theano import pp\n",
    "pp(func)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Matrix Example\n",
    "\n",
    "This example will use matrices, one of the main uses of Theano.\n",
    "\n",
    "In this example, we will use T.dmatrix to define a matrix of doubles (floating point values)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 216,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "a = T.dmatrix('a')\n",
    "b = T.dmatrix('b')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Again, we define the Theano function, and the Python function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 217,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "func = a + b\n",
    "pyfunc = function([a, b], func)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 218,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[  11.,   22.],\n",
       "       [ 103.,  204.]])"
      ]
     },
     "execution_count": 218,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pyfunc([[1, 2], \n",
    "        [3, 4]], \n",
    "       [[10, 20],\n",
    "        [100, 200]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The result is given as a numpy ndarray:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 219,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "result = pyfunc([[1, 2], \n",
    "                 [3, 4]], \n",
    "                [[10, 20],\n",
    "                 [100, 200]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 220,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "numpy.ndarray"
      ]
     },
     "execution_count": 220,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can create matrices composed of:\n",
    "\n",
    "* 32-bit itegers (i prefix)\n",
    "* 64-bit integers (l prefix)\n",
    "* 32-bit floats (f prefix)\n",
    "* 64-bit floats (d prefix)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Computing a Derivative\n",
    "\n",
    "Consider the equation:\n",
    "\n",
    "$f(x)=e^{sin(x^2)}$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 222,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "x = T.dscalar('x')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 223,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "fx = T.exp(T.sin(x**2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 224,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "pyfunc = function([x], fx)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 225,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(0.602681965908778)"
      ]
     },
     "execution_count": 225,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pyfunc(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Does that agree with what we would compute directly in Python?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 226,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.602681965908778"
      ]
     },
     "execution_count": 226,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import math\n",
    "def eq(x):\n",
    "    return math.exp(math.sin(x ** 2))\n",
    "eq(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Yes, they do agree. \n",
    "\n",
    "Let's plot the function using matplotlib."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 227,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "from matplotlib.pyplot import plot"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 228,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x7fe7e6f027b8>]"
      ]
     },
     "execution_count": 228,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAEACAYAAABI5zaHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJztvXuQ3Ud5Jvy8o5FmJI0kS7IlYwtfsbkajJeAwbCIWwKk\nwCSQ5VabLF9VlqVgIbALJJAt7CqWIqnvyxdYSLHeDzvBBUvApMBbBtYflyGFIeZiC8uWje+yJFuy\nLUsaaXSbS+8fPe3p0+d9u9/+Xc5l3E+VSjP969s5c87Tz+953+4fGWNQUFBQULC0MNLvCRQUFBQU\nNI9C7gUFBQVLEIXcCwoKCpYgCrkXFBQULEEUci8oKChYgijkXlBQULAEkSR3IhojopuJ6FYiuoOI\nPiPU+zwR3UNE24jo4uanWlBQUFCgxWiqgjHmBBG9yhhzlIiWAbiJiC4zxtzk6hDRGwCcb4y5gIhe\nAuBLAC5tb9oFBQUFBTGobBljzNGFH8cW2hwIqlwO4CsLdW8GsI6INjc1yYKCgoKCPKjInYhGiOhW\nAHsBTBpjdgRVzgSwy/t9z0JZQUFBQUEfoFXu88aYFwLYAuBfE9Er251WQUFBQUEdJD13H8aYKSK6\nAcCLAPzEu7QHwNO937cslHWAiMpBNgUFBQUVYIyhnPqabJlTiWjdws8rAbwOwLag2vUA/nihzqUA\nDhpj9gkTHPh/n/rUp/o+hzLPMs9hnWOZZ/P/qkCj3J8G4B+IiGAXg2uNMT8kovdarjZXGWO+S0Rv\nJKJ7AUwDeE+l2RQUFBQUNAJNKuR2AJcw5f89+P0DDc6roKCgoKAGyg5VBlu3bu33FFQo82wWwzDP\nYZgjUOY5CKCqfk6lwYhML8crKCgoWAogIpimA6oFBQUFBcOHQu4FBQUFSxCF3AsKCgqWIAq5FxQU\nFCxBFHIvKCgoWIIo5F5QUFCwBFHIvaCgoGAJopB7QUFBwRJEIfeCgoKCJYhC7gUFBQVLEIXcCwoK\nCpYgCrkXFBQULEEUci8oKChYgijkXlBQULAEseTI/ZvfBM49F/jlL/s3hz/8Q+Dqq5vr7/LLgS99\nqXr7Zz8b+P7389rcey9wzjn6+rt3A69/va7u3r3AW9+arnfwIECKQ05f/vL0+/NXfwXs3y9fn5oC\n7rpLvv7jHwN/9mfy9b/9W+CRR/hrjz0G7Ol6ovBivx/7GH/ts58F3vc+/trMDF/+F38B/Omfdpff\ndRfwspd1lz/xBPDud3eX33ADcMYZ/BgcDh/m5/jYY3y5Br/6FfDww+l6u3YBBw50jzE3F283Owvc\neKNuLj7+/u+BS4LHF11/PTBwp5n3+DmApk3Mzhpz5pnGvPe9xrzmNa0OJeLkSWMAY1772ub6BIy5\n9NJqbefnbfs///O8dtdcY9vNzOjqf/e7tv7hw+m6X/mKrZvCLbfYekeOxOsBxrzhDek6f/3X8vV3\nvCM+p9e+Nn4dMObjH+evveUtctu3v12+tnGjfA0wZseO7vL16/k2X/gCX37DDXz5f/7PfPn119u/\nNTefxx7rLPvwh7v72LuX7/cTnzDmpz/t7vMVr+gse+wxYx5/vLveS17SWXbuuca8852dZaOjnXP/\nwQ90n8MPftCY48cXf3/b27rbAcbcc0+6r6pY4M4svl1Syn1yEti8Gfjc54Bf/CKu1NrCgw/a/yUV\nlwunBmZnq7U/csT+f+xYXrtDh+z/jz6qq//44/b/UEFxcK/p6NF4PTf23r26OaRw/Hh6LAnufYzh\niSf48oMH5TannCJfO3GCL3fv30MPdV9btoxvI/39pdclqdA//EPgHe/g6953X2c5d/d88qT9P/yc\nfOYzwDe+0V3ffQ4d3vhG4NWv7q53772dvz/wAHDnnZ1ls7PA3Xcv/r5iRXc//+W/AG97W2fZ5z8P\n3H774u/Se5P7HWsbS4rcv/c94C1vAcbG7K365GTv53D33cBLXgLs3NnMbdrUlP1/erpae0da3O1x\nDI6sNaQGLH4JNeTuSF0iQwdHxto5pBD7e4wmniY8Nla9/9i4I5FvoCPCEG6h5xZ86XVIC1tqgeXG\nDvty9kfYF/d3c5YM9zmZmOguC+2eX/0KuO227nrcArp+fXfZxo2LP4+P2//9v8+11wLf+lZ3O1+s\nFXLvA376U+AVr7A//87vALfe2vs57NkDPO959ksbU2xaPPIIsGpV9b7cl0NDulw77aLiPtih0uLg\n+owpaf+6hoA03nyMZCXF66Ah9yrjxu7IJKJ2ip4jf2mxkMql9y1HmLh5hF4617erw73u+fnusvDu\nhXsdo6O8v758OV83hP8Zlz4H/nwLufcYx48D27db1QwAF18MbNvW+3k89hhw2mnA057WjJ2wbx9w\n4YXVyd2pp1zl7whVq5pdfclK4Opqyb3ul8Z9MSUlDKTJPXU9hqrkzpETsPgec++19Bo5CyI2D81i\n6SARdhPkHr7vHLlLc+UI3x/DzcV/H+uIhNy7oLaxZMh9xw7g/POBlSvt7y98YX+UuyP3DRvStoMG\nhw/bhWJmRkecIY4csbenVck9V7mnCNvvswnlrlGYmn5i9gjAE492LrE5VrkWU+4SnGINX4cjuPBv\nkXo/fEjKnesjRu4cGYd95JA79zfjyN0vC/vX/N2r1O0Flgy5b98OXHTR4u9nn22tiKb8Wi0cuW/c\n2Ay5T09bL/KUU3SWR4gjR2yQOZfcHVm3Qe5ala9R7o5YYkTn2sfqpL6YbaW5xZSiNKeYck8RXUiq\n7nfN302Ce1/bUO69IHd/UQn7ctf8OgOX8ihgyZD7bbd1kjuRzXe///7ezqNp5X7kyCK5V7FmqpL7\n0aPAqafqF8c2lXuM3DV13LWYBZLKidYobImMY8FaRyZc/1WUu6S4OZIC8iwVCVIfucqde//CeTRJ\n7m4OMeXu6sQ+H+7vNGikv2TIPVTugLVpwhSptnHggCX2DRuaScU8cgRYvRpYt646uW/aVE25n3Za\nnnIfG8sjd61yj216cdc0yj3Wj/vyVrFWJFvCIWZxuHG1ZOePk6PcpXE05JVCTkBVUvnSHOood05t\n+2WcLRP2xb0/YRzAtS/k3hJ27ACe+9zOsvPP7869bRtTU8Datc0pd2fLrF5dLWBTR7mfdlqecl+/\nXm/LjIzolXuMuGPkGPYTI3c3hkRyGnKvQpAxopbI3Y3DvZ4U0UnKvQlyb0O5h31wr0+j3Dlbinvt\n4Xjc+xaO5/qu8x62gSVB7seOWZX89Kd3lp93Xu9tmakpq7LXravmkYdwtszKldWyRqanLUmfPJn3\n4XPknqPcteR+7Ji1mbTkHiNl92WN1dEod0eu0iLB3cJr28Ysjtidh7SgSEQdG0siIIncU1YTN5/w\n/eXmn0vuIaoqd9e3P0fub6pR7iFc+6ELqBLRFiL6ERHdQUTbieiDTJ1XEtFBIrpl4d9ftjNdHvff\nbwOo4e3Sli3ymR5twSn3NWuaCeZOT1vVvmpVdeXu2ueod0fW2gydHHKfmbHvj8aWWbmyt+Qu1YmR\nkmsrEUDsyx8jj5RyzyF3qY00fuyOKCRtqS43/1wbKhyrquceU+5LldwT+/IAALMAPmKM2UZEEwB+\nTUQ3GmPCY5b+2Rjz5uanmMZ991kLJsSZZ/aW3F264qpVVm1zhynloq5yP3nS7sRbvdqS+9q1unZH\nj1p1nUPuGzfqyH121r4mjXJfsyZuy2jIPeWJA3pyn5npzhlPKXefWENyitkiVci9akA1LI9ZDZIt\noVHuUtaOfy1WVtWW4RYV7jUuJXJPKndjzF5jzLaFn48AuBPAmUzVjPh6s4iRu+ZUuaZw+LAlI6LB\nUe4nT1oyWrkyL93NWSfaXOpc5Z5D7inlPjamU/cxXz5F0LFAoJbcueuxO4JcFV6lTUq5a6w8yZaJ\nkbX29dZR7n45Ny5HylWyZYaW3H0Q0TkALgZwM3P5pUS0jYhuIKLnNDA3Ne69lyf3zZvtGSnaI0br\nwvntwGAp9xUrLAHmbII6fty+Fm2bkyftXYFWuWttGQ25r1yZTnNMLQApfz+m/nOUe4gYeeRmvmja\ntEnuIblxyj3H7uH6zFHuXD+cVROzZTTvw9CT+4Ilcx2ADy0oeB+/BnCWMeZiAF8A8O3mppiGpNxH\nR21QsKlTBVM4dGjR9mhauVcl9xMn8sndGEtmExP6Ns5Hb1K5z8zY156yZcbHdQtAKqVy+fJq5B5T\n9YBOuTehwmOQFLPUV8yWCUlYIrdcW4aDJr2wKrlz6ZGabBlN34MAjecOIhqFJfZrjTHfCa/7ZG+M\n+R4R/R0RbTDGdCUDXnHFFU/+vHXrVmzdurXCtDtx333AM57BX3O+e5hJ0wZcMBVoXrnXtWVyyH1u\nzn5hVq7U2zKO3HftStednbWknSJ3DSm7OrG005x+JNLRbJuXCCBG7nWUu4Z4U20kEq+i3OvW1dgy\nHHLIncugiSl3TZpjG+Q+OTmJyZrH2qrIHcDVAHYYYz7HXSSizcaYfQs/vxgAccQOdJJ7E5idtYRy\n7rn89V4GVX1yX7OmGXJ3ee4rV+Yf2wtYch4byyP3KgvCzIxdgDQW2MxM2koBFheBuqp8bi6t7lN1\nXGCau57y9DW2TFOeey65V1HuIXLILWaPVLVlNOfgVLVlNHcaVe6kUgiF75VXXpndR5LciegyAO8G\nsJ2IbgVgAHwCwNmwTwe5CsDbiOh9AGYAHAPw9uyZVMTu3dZ6kY5kPeOM3gVVQ+XelC2zalV95b5i\nRZ4Kd2205D47qyd3jU/u10vZMlpVriH3WEBVen1zc/a9qmLLNJmzDvRXueeQe0pBO4Svh3t9OQFV\njty5nayxOiGG1pYxxtwEIHrgqTHmiwC+2NSkcvDQQ8BZZ8nXN22qpnirwGXLAIukNDubfhBEDMeP\nW9LpZUDVb5OzIKQI1MGRbWrxyyVuY/gvuiPu2OtPefcnT9rsIYmgx8aqkXuV3Y11PPcmlLuWAKW/\nhd9GKmsa3BgaUh5mch/6Haq7dsX99NNO6x25Hz1q1R1gP9hNqPfjxy1x9NJz99V+v20ZDbkvX27f\nb+nL5Yg7dWsdI/fYXFLkHvPVXXwjptylTUOx3Hhtm0FX7iFyDjTTjluXuAu5t4RBJXegGd/9xIl6\nyr1KtkzbnruGtF29VavStszoaDzTRbtIxBaAGPk7WyYWUCXKV/0SaVRJreyH586dlsiNlzNW1cO5\nqnrummMYCrm3hBS599KWCcm9bsaMS0kcGxsOW2bVKl2KW5PKfW4uTe4pVW6M/WLGcuFjnrzGlpHI\nP3UN0Kcv+mXSgiAdEdBmnnuKUGPKv6mTFrXkrmknBV0LuTeM3bsHV7lXPcnRYWbGnpczMlLPlhkb\nywuo5toy8/P2S5jaKOSQo9w1iruucp+ft+/xihV8nRT5p4KxMfKPkfv8PH9NQ+5aJd4Euaf6SAUx\n69oymgVAG1DVtKtSpx8YenLX2DKPPtqbuYTkXlVtO5w4sZgFNMjKfWYmTbBh/ZQH7uo1Ycto/PRl\ny2wfkj0yMiKPoVHuueTuFhQuC6fftozk54fkxvVRN6Cao+S5bJncRSW2+Sm8Mynk3jBS5L5xo33I\nRZM5qBJCcq+qth1cpkydvqqS+/LleuXugppacs/13DXKfXQ0HtCMjTc3Z8l9dFTOY4+Rvzvfpoot\nIxH4/LwlJ+51zc3Jc3GvRevTN6ncJXKvo9wH0ZYJyb0X2T5VMNTkfuyY9bRPO02us2yZTWFr4qlI\nKbSp3HPPhnGoq9w1bdzWfQ25uy9AjAwdmrRlUrZJrI8U+WsCqrnK3R+TuzY2Jiv3HCsnR3VLSPn8\nnHLvdbaM1iLSHK0QllVJZ+0Fhprcd++2O1BTO9R6FVRtWrk3Qe5VsmXcJqYcm0VL7r6FoyH3Xtoy\nkvp3RNtLW8Ynd065x4Kwy5c3p9w1AfKUn68NqPYiW6YJW0Yi96LcG0TKknHoVVCVI/dBUe45Oeuu\nTRvk7iwcSQWHdcfH41/6Xir3ZctkQk3luUtKuwq5z8/bucZ8+hwS515XE8pdm1veC3KsastoMn4K\nubeAQSf3lSub89yrkLtLpcxNa2yT3HOVu+bER41ydw/Y4L6ATrnHyLsOuaeUO3fNH1Or3Ofm7F1s\nrufOLRRNpEJqVTrXnsuRBwYjW6aQew+QQ+69yJhp05bJSWV08L/sVTx3R8CpL48jd60az6mbyofX\nKvfRUZ4ogbi/7beXrmvIPTfPfX4+7rlL5D46av/m3DVuQY2N766HZVp/Xuuvx4KW4VgaIo8dQFY1\noBp7DYXcW4CW3E89dfgDqo7cczxHR9JANXInkgnRR04qZBXlriF3iXhdnRh5O1JMkX/sumS7+Ndz\nPfeRkTzPPXYHEiNxSbmHxyK4lFBj+F2nTWbL1CHMJrJ0cuZWyL0F7NplH4KdwoYNwIED7c+nTeXu\n8qxz1Htdcgf0hJ3juTsy1nruGnKPLUKOvKWAqYb827JlJI+8qi3j2nBEy5F4jNzDMdzdRHiOz/y8\nfe80ip5bCGKZLGGfTT51iXtYhyaHvZB7D6BV7uvXxx/m0BSOHWs2oOp77kC+7+4yZXLb5pJ7Tp67\nX1dzR6Dx3GOZLpo6WuUes21WrLBfbongOAJ1fUsblXKzZdxCJ9ky0oKgVfS+zReWc+TOlXP9xhYB\njecfIub1pwg/nEsh9z5BS+4bNvSG3JsOqPrKHcgnd5+kc48fWL7c/ty0cne2TEq5z88vHmnQhC0T\nI+8c5Z6ybXIUuLseU+EjIzxp5ir3KraMpNzDOeXcFXD95ih3KdAazl3Tn2YR0LSrsxC1iaEl98OH\nLQlt2JCuu359+7aMIypHikCzqZBAPXLvhS2zbOHU/9iHXKvctfVSlktYpyp5u8Uh1l5S2bH5aYg6\nR4U78u21cudy610f3EKQq6AdOHLP2XiUumMI25ZUyD7AqXaNB9cL5R6qdmAwlHuVPHm3iQmIk6Zf\n31f6sfpa5a45VkBbT2vLxGwXzSYn7rrmrqBJcneLEEe0TSn3sH+OsP35cP68JqAqLRiubwlalV7V\ncpH8+ELuDUFryQD9I/cmz5YBemvL5Cp397SpVP0c5d4UuWsDqlVsF7//XFvG9+O5azFyl3LjR0bi\nyj0nFZIjd65/yXOX1D9X5r8f7mfJxw/rhmib3Itybxmpo359OFumqUOIOEjKvUlbJmeXKdBbctd6\n9FWUexMHh2lsl5RtUyWbJnYtReCxDUmSx91kKmTMlsnx3KvaMtz7HVPSmgyXXFumkHsfkKPcx8ft\nB7KOik6hDeXO2TJVUyGls8pT7Zomd3/TkXZzUhO2jMZ2qZPnXoXcJQ/bb8epcMliSS0IuZ57LKCq\n9dzD91yye9w1v0xry0gBzqLchxQ55A60H1SVyL2fAVU/FbJN5e6sFkCntDU7VMMMF+muqwlbRqPM\nNQHVKuQe24XadEA1FvTUKHdny+R47hrlztk6rozbMMVl68T+d+3CsqLcBxS55N62795GQLWfnnuV\nVEhNfe0OVT9nm0sHDOu1meeeCoqmdoZWSZOsQu4S+bpr2sCpK+dIOOa5aywfKaDKLQKxtMsYcTfp\nuZdsmT6ginLvNbm3YctUzZbJIfccss6tr1XufpA2VjdHufc7z73Ja20rd4mYc7JlJCLm8ty5rBpu\nYZcWh9j/fn/cnUhqoeCOYfD/d9e4z04/MZTkbkw15d5rW6bpgGqvlHsVctdmy/gB1bm5tN0C6P30\nOsq9braMdN23l6qSu9Y/rxNQ1VgqMeUe60Oj0rky7Vgh4TtC1oyhWShSAeCi3BvEoUP2j752rb5N\nP22Zqlk6w0Tuuco9dSiZ7+PHLJymAqp1s2XqErhE1LkbklIBVQ0Bu/ElqyScL1fXfeY5L12yajhy\n19wlhIQv9ScdhZAibq5d7P9BwVCSe65qB/pjyyxfbj+g2iyVEE167jmHjrVJ7qHdkiJkTb2YKgf0\nAVXNDtUmbZmU0o5l0kg2SCzDRkvi/hiagGqsLmeFSOScau8WjNTiItk82jL/9RVy7zGqkHs/bBmg\nnjVTN8+9araMT9ZVdqhqsmBSdXtty2hTJXNtmdS1mNKu4rmncuPrKPccW0aai2SFaNpzZe7/sExr\nwaTKJHL3bZ9C7g1i927dUb8++qHcgWbJvU6eu1PuGovIP34gNxVSa8sAecq96oM4AH1ANaXMNcq/\nii0jWS9VPfdYQLWuctcGVKW55ARUw2wZrk+pnmQHaX34cMxUHf//QcFQkntV5d42ua9c2V3eNLlX\nzZaJKU+uXS9sGY2XnqqXslxcX5psmCo7UFPX2yJ+iZCrbGLK8dy1xw/keObasfwyn1C5elUsGG4u\nxuisG///QUGS3IloCxH9iIjuIKLtRPRBod7niegeItpGRBc3P9VFVPXch82WadJzB/TWTB3PXbuJ\nKVW3ii0jLVwp6yal/lPkr8lzT6nz3GtSznrVVEiNco9ZLVJdycKpElDNsWqk4KnWlon1FRL+0JI7\ngFkAHzHGPBfASwG8n4ie5VcgojcAON8YcwGA9wL4UuMz9VDVlhk2cm8yWwZol9w1KjunbpOeuzag\n2naee66vHgvE5qZCcuTmxomVa62WWGaNZiGI5blzFkyuLcMtYpIqT5G7ZMto7op7iSS5G2P2GmO2\nLfx8BMCdAM4Mql0O4CsLdW4GsI6INjc81ydRVbkfPNjOfIBC7pq0RUCv3JvcxKQ9OCzmyUupjmH7\n8Lo2z51T1CnlnmvL5Ch3rjyl3DUWTsyWCetxilwq88ldWiyksvB4g1xyl97bfiPLcyeicwBcDODm\n4NKZAHZ5v+9B9wLQCNwGplzlfsopTz3l7mfLAINhy2g3PGk991zlnmuruPZVPfmUAo8d0Zvrn0vK\nOtbGvTcaRR9T7lrPXbJ7Ypk5oQWTKpMWkCq571rlPojkPqqtSEQTAK4D8KEFBV8JV1xxxZM/b926\nFVu3bs1qf+CA/ZCuWZM37imn2M1Pxuge8JGLQST3QVXuviKP+eQ5WTV16vjk3etTIat69U2lQuYo\nd0fYrr+wrsZzjyn3tm2Z0dHOz7+WuKvUqYvJyUlMTk7W6kNF7kQ0Ckvs1xpjvsNU2QPAN0q2LJR1\nwSf3KqhiyQD2wzM+bh/Pl7OzVYtBDaj6i4N2I1MuuVc5FTJVtx957lWyYfzrXP9tEX8/NjE5wnbt\nYn1wpCv1KxFoaFmlbBmf8KVUyOPH0+NyqZDhfP3dvm2Qeyh8r7zyyuw+tLbM1QB2GGM+J1y/HsAf\nAwARXQrgoDFmX/ZsFKhK7kC7QdVebWKqmuee037YdqhqbZlUnnsbqZJtkzt3HG6OcpcCqjHCTuV9\nu7KczBpNQDVmy2iyZaqU5QRdBy2gmlTuRHQZgHcD2E5EtwIwAD4B4GwAxhhzlTHmu0T0RiK6F8A0\ngPe0NeEqmTIOLqh69tnNzgkYHltGcxRCm9kys7OL71NOtkwTAdUYOadSITWpknUInCPdWLvR0cVd\nksuW2fIqnjtH4sYsEpb/eZOUe0zla47sjdkyIWlrbBlpsajiw2vaud/bfNJbFSTJ3RhzE4Blinof\naGRGCdRR7m0GVYeF3HOVe/gF19RvWrk3GVCtotxTAVdNnnudHPgY8fvk7qtYvz9H1jk7Ud3BblxA\n1Ridj87ZMpzKTSl3rS0TLgLutbtFMJXT7p9dExJ3zM5pw5ZpAkO3Q7WuLdNWOmTT5D47u3hL6DAM\n2TK5qZC9ODhMG1BNKfMqnn3smm9z5Ga+SFkr3ILgCC6W/661X2LnxaRsFTdHbjGRUiG1tox0TEGu\nAuf66kdAtQkMHbnXsWWGSbk71e5n9gxDtkwbyr2pgGrKdqniqaeup1R97OTH2DWuT2lB0Hjx3CIh\n9cMFSbkAZp3jByQvPWbL+GVVSNp/fbHgLNeukHsDGETlbowl8CbPlgktGaBaQLVKQLbt89w1Xnob\nT2JK2TIa5T4IAVVJWUuEGrN/tH63P66U3igtNByRa2wZibRzMmhCco/ZMpLiT92pSJZXvzFU5G7M\nYCp3Z384n89Hk+TeS+Ve5wHZWltmkA4OSyn3QQuoSjnkuXaN5IFLJJ6jxomaDajm2DJcPa0t44+p\nPTisKPea2L/fkuXq1dXat5UKKVkyQHVyD3Pcgfrkrslzd194t1BVyZZpypbJ8eYl4gXqK3ftDtY2\nAqohOcb6lKwcaRznxWuCmX7/UpBUc7cg2UDagKo2W6bKrlWtneMIP2bdDAKGitwfegg466zq7duy\nZdogd0m5t53n7sjXef3DEFCNEbcxuk1MGttlUGwZSYlLdk1McUtBz1j/2uMHpD5yToXU2jKagKom\nMJrj1bvsmkLuDeDBB4Fzzqnevi1bJkbu4+PNeu5t57n7RA0MTkBV47lzxOvbA6mAqsZ26fXxAzmK\nX7JrUvVzSF+7MUlS/zEFbUwnYWqIXGvLaD33ki3TJ9Ql96Wg3NtOhfQf1AEMv3J3xO364RYA7Q5V\nyZbRnvzIBTNzM2Ji1yTFXUWJp+qnCDtl4UiLD0fari5XJlk1bdoyXDsubtJvPKXIvR/KfZA8d60t\n00vlHlPkvu9fldz9fjR56m1ky6SIn/PVJSvFv8YRZ4ystUo8FVDNsWUkzz21aMTKuCCrNluGG7cp\n5T5oO1SfUuQ+TAHVfqVChuSeyn4J2+SkQqZIu4lTIUPlXnWHatVsmbZ3r2ptkxzlLi0sVWwZTd2Y\ntSKRbyygqrVlwkwYaR4aO6fYMjWxc2e9c2GearZMr5S7lrBd3aZsGU2w1O+nDU89db0fAdWYQs9V\n7m3YMhp/niNyrUrnxpbuGFJ57uEC4OIC4d1DIfcaMKa+cl+1yhJVDkFq0CtydySVUtIOw2DL1Amo\n+l9srS0jkXMqz12TbdMrcq9C4imy1tgvVW2Z1EIQU98hkce8+ZSl04Tn7h/lEOt7EDA05O4U9ymn\nVO+DqB313ivPHchT7/0g96aUu+akSY26z7Flhkm5cyQu2S+5Nk4sMBuzWsKy2EIg2TJcUDRVpg2o\nNpEKqe17EDA05O5Ue92nKLURVO2Vcgf0vvv8PE/Ug6bcYwHV1CKgCZZqA6qpgKlG2ddR4CExSAQb\nu8aRZGzzMStXAAAgAElEQVScKspd6kcibK0/H76emN3C1dMGRlPeeSH3PqCuJePQRlD16FH+XBmg\neXLXKndHuP5iOEzKXaPKc5W7pMxTAdM6yr6KAvfbxQKqEvnGFHcTyp0bN+fI35jKTxG51papmwrp\n3wUUcm8ZDz7YzEM22rJlpCMRxsctoXKKL4a65B5mygDtbWLSbjhy/WuzZVJ9VlkAUp56rjJ3Y7Tl\nq+dck8i3jufO1edIn7M8tAHVlG+utWU0Pny4Uaoo9wFAU8q917YMkSV4/7B/Dep67qHfDvROueec\nCllHuef46Zp+Uso8ZdtwRFx1g1NK1ecqbm4cztLw6+eQfixbporK5+YWs2U0yn3ZMvt9TJE7d7fg\n/+52PEvqflAwNOS+c2dztkwvA6pANWumrudeh9zDw8b6bcv0MqA6N9e9GUVL/nWUe8xXz7VltMo9\npfRzbBlHbj5xSn3k5LmnsmBSqp+rJ5Ey11eYCiktAIXca+CBB5qxZdpQ7tPTaXLPVe5N2DK9UO7u\ni+Kr4149rEOTw+4r9xQ5j4wsPpKN66OtbJmYr55D/JLa9wlJ49Fz9kesvptPqIq1tkyOBVMnW4YL\n2qby3MM6xvDxgULuFWEMcN99wPnn1+9r2JW7ltzDc2WAdnao+kocyFfuqQPBYnNoSrmHQVeJaJvO\nlqlK/DFSlki/aeWeslVidTVnt+faMlWtmqY895LnXhH79lmCrJPj7tBWtkzT5B7z3Nu2ZXKUu6/E\ngd5uYtIES7V1Yr58jifPtW2D+CVS5gg15rnnKndtOUf4rjyW556yUepkyzSRCaOpMygYTVfpP+69\nF3jGM5rpq62AauwBIlWVO0fu2mN/uWyZNvLcc+v7yj2m8pvaxJQTUAXkoGjoyfspplVJuk6wNUXW\nbSh31797oAvXT4ycXXnOwWGSIm+K8LXEnbJ8BpHch0K5N0nuw2LLHD8+HJ57ro3TlufeZkZNzJOf\nn1/0YZsMqEpEHV7jSFZL+hrlnmPLpOwSV+7I3fnzWjKO2TKpgKpk3yxl5T405H7BBc301etUSKBZ\n5V6X3JvOc6+i3LXkrtmhmkPcMVtF47lz12PX/OuxoKl0jSPS8JqkuLV2TawfKaAas2ViBOuX+4tk\njIxzVHpKXbfpuRdyr4ii3BfRi1RIjqyls6p9gnX129ihmvLcHXFLaYyunyq+fKyPFLn71gtH0rnW\ni9uEk6PQq3juOYqeI+KYyo+p75Cgq9gyMdWf67lLqZCF3BtC0+Tej1TIJgOqbWbLhE9icl+wmDce\n2jKScjcm72yZHM9dUrlNqPvYda3qb+qaI6nwZEL/WhOee6w8Ni6nxlN9p+wWTT2tVROzZSR7yMUZ\nXKyhkHtDMKZZcl+3DpiaavYP0cuAaq89dyButXBKX1oI/C9Gqm4uKUv1cgOqknKva8s0HTQFZNsk\npsQ19VMBVclqSSlvaS45Hnk4TpVsmbC/lD3kbKRC7g1j/377Rm7Y0Ex/y5YBExOW4JvCoAZUqzzJ\niSP3mFefo9y5tMmmAqpSvVBZV1HuPoHn2jJVA6rSNUdm3HgaJZ6yU8JyjdJP2SApWyaVbaMlfE3b\n0HLR2kOunfRah5LciejLRLSPiG4Trr+SiA4S0S0L//6yyQk2qdodmgyqzszYP2pIiD6aToXspecO\nxFMouU1MmmN8gfqeu0/cUn+hcuc8d41ylxaIJsg9J9gajqdR6FUCrbGAqsYOiqn8mG8eK8tZBLQB\n2tHR+K5aDbkPqnLX5LlfA+C/AfhKpM4/G2Pe3MyUOnHXXcCFFzbbZ5NB1WPHrGqPnTM/zKmQQNqW\n0arxfih3bdZNjufeT1vGHy/miWs895iNI1kq7q5BY7VoffsYqaZsmZglpOmPC5aGddxBYSlyH7oH\nZBtjfgogpXNrPkJDxp13As95TrN9NhlUTVkywOB47tpNTDmLQk4qpIaMubqagKrUX9Oee1u2THjN\nJxrJc4956LnKXVoMNPVjVotf5sivakBVIt+cxSKmuCXrJlTuPtlz/QwKmvLcX0pE24joBiJqlIp3\n7Gie3Ju0Zdoi97rKvamzZYDmAqqcLaO5I6ir3DWee6xOzJaJqXq/bZPK3ffcU/YIELdIUgFYTTlH\n+jFrJHyvYnXrePNSvdCWyVkUuIPDBpXcmzh+4NcAzjLGHCWiNwD4NgDRSLniiiue/Hnr1q3YunVr\ntPMdO4BnP7uBWXpo0paZno5nygDNpkLmeO5NPKwj1S6s7768vqL162pz4tsIqMby3GO+fB1bJnai\npK+QXaqds/diAdWYLRNbEKoEYDmF7n72y2O2jETuGuKVvHT3eeQWBvc359pqbJlB8NwnJycxOTlZ\nq4/a5G6MOeL9/D0i+jsi2mCMeYKr75N7CseOAQ8/3MxpkD6GQbnXtWU45e7Iz/+ShagSUB0d7a4f\npim6ujkB1dTpkeHYkl+ec0SBpNwlayfHlnHKLyRwosUdm64vbUBVG/BsUqG7n/364eLBEbbfnlP5\nGsLXWjCxgGpVcp+b650tEwrfK6+8MrsPrS1DEHx1Itrs/fxiACQRey5++1ubKROSR100qdzbIHdj\nmjnyN2xPlFbvdW0ZQCbjqgHVup57Tr58W3nuPoHn9MsReMqWqeuhc1aHG1sbgJWUMvd6JX89Rvia\nRUATZNVsYvLLuOD00NoyRPQ1AFsBbCSihwB8CsAKAMYYcxWAtxHR+wDMADgG4O1NTa4NSwaw5H7n\nnc301Qa5OxLk1HXOkb9r13aXO1uHWzjc2HVsGUBeDOqkQmrJnbNFUp67RrnHjh/Q7FD1r4fqnGsb\ns1gkW4azR8JyrUKPLQaubViuIVNfuYekqg2oar39mAL322oUv6sXs4GGjtyNMe9KXP8igC82NiMP\nbQRTAWvLDLJyl4KpgP7IX86Wce1ji0OuLSPV15JxUw/rkOo1odxTxw9olDt3vS65N+Wtu92XzjLi\nlDjQbRuF5ZItI3nu2oCqe7+luwFpYdCU+ccP+wuAP6YmFbI8rCMTbaRBAoOfCikFU4F6tgxQndy1\nG5OA5m0ZreeuIXcuoJrjuVe1ZTRtc/zwVBtpoeDslNBDlpR7jPTDRcUnbM5z1/rrKUWeslJi/YWK\nm6vDpUL2IqDaBAaa3G+7DXje85rvt+mAatPZMhIxA/XJPZXrXsWWkQKqqb6lesbEg5gOTSr3tvPc\nuesxiyWltmNtpHEkGyfsK2bXcF58zFbhyjTjcQQtLQKcTRKOHc7d5dznZsvE5gAM1kamgSX3qSmb\nKfPMZzbfd9OpkL1U7nWOH9C0b8KWiSl3jefuByHd+HUODnPEPTLS/RQh90X3FWVOrrw2z5273oYt\nE/PcQ2LTKGnuzsAnttTGpJAocwKq2iCtNI5UNjenvzMIy7RzGAQMLLk71e5/eZvCoNsyUhok0Iwt\n03a2jFRfS+7anay5yp2r45Ovu96Ub+7Gb8tz58iXs018b935665+TLnH7gzq+OiurlZpp6wVbeDV\nH6OqnaNdFAYBA0vu27YBF1/cTt/DHFDth+eec/wAIJOxlty13jxHzBpy9wkrVP8pz70tWybHJ0+1\n4UiZa5NS7pK3rg2SSkQJpK0VTiHHLBGtJRQq8CrKvZB7Tdx6K/DCF7bT98qV9v/cjUUcBlW5S+mO\nbQRUOc9dq9y1KZNNBFS5OlrlLtkybQVUY0TNETLQSXQatZ+r3KsESbkyqW5KpefYMilVHrsLSC0K\nhdxrok3lDjQXVM0hd22wJZUKqfHcq6ZChk9icm1ybZk6yp3beeq+WLF6Vchdo9ybOBUy1VZS21WC\nsNo55ip3ifxSC4S2biqgypVx9kpssUjZMtyc/WyilJ1TyD2BkydtGuRFF7U3RlNBVU22zOio/YCk\nznVx6HcqZM5pknUCqtp8eKL0BiU3borcOeWd47nHlL90tk2VhUGjwrVtQlL1SZAj25S9o7FlYiQY\n1tV45H6f3OKSY+nEbBnuaAG/HteOe62DgIEk923bgAsuSJNmHTQVVNUodyDPmhkmW6YXyl2qq/Hm\n63ru7svKkRLQrufOqfDQc5eUdczKCUk87IsjML9+zJaJWTshuWvtEE2QVlLzUltJ8ceIW7JzpPer\n3xhIcv+XfwFe+tJ2x2gqqDo9vejhx5BD7k0FVJtKhYzZMrmbmMITJDV2i9SnZrHQ2DKhsg+JMbY4\n9Dqg6pOp5IlrPfeYcpdsGc4+kRaImOeeIm0t4cdIOubra0maU+7FlqmBn/8cuPTSdsdoSrkfOQKs\nWZOu15Ryz/Hcq2xiyn3Ih7SJSUPuWrsFkInb70/yy3MDqjmefCrPPcyR15Cuf02yRqT+OJKJee4p\n5Z5jy6RIN1wIcpS2X4+ba65Vo7VXwjGL514TvVLuTZH7xES6Xj+UexVbRiL33FMhNbaMqxv2LWXg\nVLVltOTsrktkmrrelnKPtZGUdV3PPabcc20ZyXOPqXytLRMj5NhiE1t8YsQdqxOe+tlvDBy5790L\nHDpkPfc20VRA9fDh5pV7LKDqyEw6k8UhtkM1FtiVHvKRe3CYRrm7uim1DcieexVbJmW7SKmOUvte\nkruklN01aUGo47mHD5DOsWUkz10i7bBuVR8+pea1C0WuLVOOH4jgppusJTPS8syatGWaVu6xs2UA\nnXqvqtw5r76tgCqgC5TG6oXknrJ4NLaL34cm26ZNcneE4R8ZkLJlmvLcwx2tdW2ZcJGp6q9L9VJq\n3l9AtJ572Ffx3Gvghz8EXvOa9sdpKqDaa+UO6Hz3OrZMzuP5tMFPQE/u2j41qZAaZa4NmALt2jKc\nBSKd2Mi10QRUNco9tRhIapyzZSSVX8df11gwkn0U886bUO6F3CP44Q+BV7+6/XGaUO4nT9o/Jmd/\nhGgqoArolHvVVEhJuTdxnnsOuWv6rGLLaDz3Jm0Z/7WkFHXMJw+JkGuTq9zDQ9U40ufKOdUdvr4Y\nCcbqcgStsWW4RSDXltEod+2iMAgYKHLfvRvYvx94wQvaH6sJ5e4sGWIfQNiJpgKqQJrc3W2nT0oO\nMXKX2jX5mL06yr1Xnnvseq4tU+fgMMkn58oBntRceY7nHmb4cMo95blLKlhry2jUt99nbpA11U66\nq9AuCoOAgSL3H/0IeNWr2vfbgWaUuzYNEuit5+7ac4tOjNyd2g/bNfWYvTrZMk2SeyrPPTebRkp1\nTF2v4senrJyQnMI2Ws89Vq7x3Lkgqb8waeqm1Hf4mrl6ksLXWDBc/8WWqYjvfQ/43d/tzVhNkPvh\nw7pgKpBH7seOxXe9jo/H+4o9IzVmsUgZNrE22rRFoH62TBXPPTegmvLcY+1Dgjam83qMdKuQu9Rf\nk567Vk2HfTvCkzx37m4kNVbMlkn59eHCVNXnL+ReASdOAN//PvCmN/VmvKZsmTaU+9Gj8V2vK1da\n60aCtDsVSCt3idwl5c4tJG3YMm157jm2i7uuJWj/S++uc0HT8Jr731e5KYL0x0vNsa5yzyHsmMqP\n2T1SGUfanL0iLUIpki7KvQX8+Mf2eamnn96b8dauteQc3kbnIFe5xwjZR+q8mtRCEbN1YhZLlQd8\nSAHYugHVXtoyKeVe9fgBzdEGKXIN24VKmVssRkY60yelIKyW9FOeOzcfyZuOef/aQGmKyFNt5+bS\nvrw0ZmzRKuQu4NvfBt7ylt6NNzJiCf7Qoep9aHPcgXzl3ia5N2nLcG1ylXuvPfcYeWs891hAdX5+\nkVRzcugluyR2LbRlHBG59MnUgiD1pQmoanz00EJJzUNS5H6f3GuLtZVUv/9eAYuPX4wp97m5+KJV\nyJ3ByZPAP/0T8Na39nbcur67NscdaNZzT/UlkTQQJ/fYeTQ5tkxOQJW7Kxhkzz2mvt2mH/cFz1Hu\noV+tIXeuv5gyBuL2i9auqeqjp+4gNKQ9Ohr30v1xYt48ZxHFlLu/KBTlnoEbbgCe/WzgvPN6O25d\ncn+qKPfUgsApd60tw/VdZ4dq3YPDUp57DvlzfWvI3Sdpd00iU59MpGsa+yWnXGO15NoyMdL27yik\nRSAnoOrbMoAuyMp57txdSiH3AFdfDbznPb0ft25QtZ8B1TbIXQrEtq3cOXLv5SammG2TOnkynINP\nWrnKnSNXQGeDxK75i0xO4FQq54gtfA0xhRvW9Um7SpBVUxbOJ7yL0Prp0sJRyJ3B3XfbUyD/6I96\nP3YTtkw/lHudVEhNnjvXJkbu4YIgjaEld8nHT5F7SLyAjrxjnnp4F8H55k2RuxtX67lrPfyUhx6W\nh+mdIdFxxAbwi5BmYcqxZaQyyYKR+vMXGmelSXcboZ2TqjMo6Du5/83fAP/hP7T71CUJTSj3fuS5\n10mFbDrPnRtL2mTVNLk34bmH1klKuYfXY+SfQ+7+vGLkHrNlYuSeq9B9pa/xt6V5couDP3eX1ZOy\nUVJl3IITU9ecLcMp8JgtM+ie+2i6Snu47z7guuuAHTv6M36vA6pHj+rq1vXcYwePVQmojo/LO2I5\ntT82Vk+5c89x5QKqbZ0KmVLuqR2uHKmG14BOcvVtpxi5pzZFSdk3Oco9x5bxSZKLG0gLQeh9a7Ng\npDKX7eKPE1Pz4VykbBkpEDsM5J5U7kT0ZSLaR0S3Rep8nojuIaJtRHSxdvCPfhT4yEeATZu0LZpF\nLwOqExP2kXwpzM+nDw5LkfuxY7JnXyWgOjYm3ylwyn3FCr1y5+4KpAeG9MNzj3nqrr10PRZQ9ckk\nnDtH7r5lwh1EFu6Gle4gcj13jkj990haODSeuyP32FhVPXdfuXN3B9psmWFW7hpb5hoAvyddJKI3\nADjfGHMBgPcC+JJm4GuvBe68E/jwh1XzbAV1bZkc5T4xYReDFI4ds8QeO4ysLrnnbEgC4mfZSMq9\nV7ZM3Tz3lG+fUv6xp0HFVP38/GLqZNguFlCVrBefYMJrkuceK4+RvvQaU743Z8uEC0zKlhkdle8m\nJMKPEblfplHumqDroCBJ7saYnwKI6dvLAXxloe7NANYR0eZYnz/9qVXs//iPuodLt4VeKvc1a+xi\nkELKkgHaVe45towxto2GsIF2Aqoaz90nSo6cU556jnL3M4Vi5B4SeEjuGh8/tpD4RBOSOLeISJ67\nZIv443J3C756jgVUU3cP3KKTUvjhwsItIKlsmbm5RcXvn6kvLQBDSe4KnAlgl/f7noUyFp/+NPAH\nfwB87WvA85/fwOg1UPc5qlNT7Sj3FLmnsmWOH69G7jHlztkyjlxHRrrr5yj38E5Cu+u1iTx3jXKP\nkX/Yv5bcuXm5dtogrfYuwSfg0FLReO6cLcMtRFxA1R+PC75qbBmtSucWFk65O89dUu5hu5DcJetm\n0J6h2vOA6nXXXYF3vtM+Tm/58q3YunVrr6fwJE47DXj88ertDx606l+DHOWeupvRKPdYQDXHYgFk\nso7V70cqZBXPXaPcU+215B4j4/BaLL3SvTd+HCJ1l8AtCBrPXbJlUpaPI8FwPM4e4cZy7bWE74jX\ntQ2tlHABkeaSyqgJyV0KxNbF5OQkJicna/XRBLnvAfB07/ctC2Ustm27ooEhm8GmTcCjj1Zvf+gQ\nsG6dru7KlZa0uK31PrS2TCwVMmbL5AZHATtfou65S/WlBYTLxtGSexhQdQHElC0T9pXruefaMv6d\nSMwSCvv1Xx+3oPjWhPt85Ch3n9zd/CTVHLNaOM9dWggcOcdUfujZa4KnHOE7Ne6PnWvLcJkx4WuT\nFgCn7pt6QPbWrZ3C98orr8zuQ2vL0MI/DtcD+GMAIKJLARw0xuzLnkkf4Mi9yh/EmDxyJ7K5/KmM\nmbY999jCEDuThlsUYtk1nHKvQ+5cfrofQOTqGJNW9ynlrgnIhsrdvZ7QOw9VvTQvTrlzJF6V3Dnl\nLnnxnI2TImygW3mHizEXUK1ry0iLAGfL5Cp3vy/prmDQPPekcieirwHYCmAjET0E4FMAVgAwxpir\njDHfJaI3EtG9AKYB9OEggWoYH7f/Dh2y/nsOjhyxbUObIQZnzcQWBI3nrslzj5G71DaWH++sGT+A\nHNvRmqPcw1iEhtw1xwc7VRlbAMKceo1yj3nyWlsmJ8tGuuar8BzPXSJxjrBduW8/xGyZ0AoiklX6\nzExeQFVL+KmAqlP5UllK3bs4kyN37s5jEJAkd2PMuxR1PtDMdHoPp95zyf3gwfw2mqBqL5S71PbY\nMTlAzGXM5KZOcotHjnL3A6oacuc2Q3Hk7b9XKduFu6713LXKPebj+9dSnjuXLSORu7SJytX3Uzdj\ntkxszHAhOH68W7lzpB0uGBrC53LT3RgzM51lbrGq4rlLr38Q0ES2zFCjqu9eRe1rgqpNBVSlPsbH\nLXlyH8Jcrz6m3Ju2ZULPPSRHoJvcq9wBcNdzbRkNuaeUu8aykUgWkFMbUyQe9uXKYxYOp9y59MvQ\nyw9tDckCCmMBKR8+zHrhrJPQcnELmEa5hwtAIfcBxaZNwL4KEYKDB/V+u0NTyj2VChnLliGSg6qp\nRaGOcnc58U157ppjgbkcfE1ANSfgmmPLhOQeErhW8Ws895iVo7Fl/PJQ6brAoVOuKXL3lbFvYYSW\nCTcHTdojV88PqMYsmHB+kuKP+fKDass85cl98+Zqyr2KLdOkcq+aLePac4uDxnP3kRNQnZnp9r+B\neuRexZZJLQCpgCpH/v51//XEsmW4RSFmy6Q89zB4W4XcuSwaTrk7a8b56FxA1H8Nki3DKXeO3LU+\nvKTmpcCob8twyl2yZfw6xZYZYFS1Zdry3DW7XusEVGPtm7RlOJWvtXCqeu6hqpaUe07ANGXbxFR2\nLFsm15bhlHvMcw/Vfkqhh3ZNOIZf13+fODUezse3ZXKVe92Aao4twyn3YssMMXrpuU9MpJW75rya\n1AmTVZV7m7ZMLrmnHtah8dy1AVXJVuHGSQVc2w6oam0ZaZGpYstw5C4pd5/c/dhAmC3j6lZR7ppU\nyNCWkRS4Pz+tci+2zJCgjudexZZJKXcNuTvCko4RSJG75NmnlDtny8QI298/IFk+2lMhq/jpUj8x\nTz18nXUCqjnXtJZNDrmn2vivPSeg6pdrPfdYQDW28HDKPaXm/YBqLBUytGWqKPfQlvE/G/1GIfca\ntkwbAVUNuRPF/ftYQBWIK/ccz/3ECT7P333ZfDJuwpZJ9addAGLKPXydnHLXBlS53bEcsYbXuOCo\npNxjOfWpNiG5xzx3LvYQ2jKp9EvOluEWEicK/DNd3LxGR3U+/MhItyL355jKlvEDrG6+UkCVu3MZ\nBDzlyb1OQDWX3DUBVe0xwrGFQuO5cwHZWLvx8e42OQeU5ZC75mEdXH/hXYAmoBrWCefDBUxD20a6\nHvad2uCkUfxVbBmNck+lSEqLkaRcOc+dS5uULKCRke4HeLixtAHVsD83Ry6g6pO5H3SNKfcwA4h7\n5kA/8ZQn96rKff9+4NRT89qsXWu9+hi05B5bKKo+ySnXlomNE9aPkbv2VMiUcg/jApxyT3nunC0T\nnh0Ts238BSZ8HalUyNxMmroBVcmW4awS6ZgGTZ675LlL2TKhGtYQuSvzVT/XnyNl7rWHfYWLgt8u\nducxKHjKk/v69fa8l1hqIYfHH88n9/Xr0w8HOXKkvnJP9dEUuafqV1Xu0iamlHIPM3qqpFSmbJlw\nvilbJhZs1aZChhaLa+cWKu4QNdefMXLQUuu5z8zwdxpVPHdNtgxHxoAuyOqrfjf3cIeq88rD1x76\n6Zxyd69hdDSeyjkIeMqT+8gIcPrpwMMP57WrQu6a8+PrKncXyJQOAAOqZ8uEC2BMuYcKty65h+NL\nyt2vw9ky4aIjkbtTgCHRcu0lxcwp9xiB51osvj0g2Tw+YYXjhLnonNJ3c5ZSRrkgaThPLhWyjnKX\nynyC9ufOpSv6rz20iKR0SX9R8OfKWV6DgKc8uQPAmWcCe/bktWlLudf13Kenq+fJxwKqXIZNjtKX\nsmW4nHiO3MM5S8rdz9LhbBluAQhJi2hRGYbjhItRmDEUWi9hQLXKqZAx+0UKeLp5SPXDcULSd/PO\nsWXc6+PuFkILI0e5h0ROtPiUJFfmXi93NyIp8LrK3b2/RbkPMLZsySP3+XmrwDdsyBtH88zWusr9\nyBF7tHAMsR2qElmvXt2dWx9T7mHQVlLu3FyqKveRkU7PW1LuMXJ3ddyCkyL3MNc/pdw1qZApW4az\nX3yS8ccKy1MB1fCYZE1A1Y8BzMx0krBfNyegGrNl3AFmvvKXyD18D3zvPFTcKc/dV/zcXUEh9wFE\nrnI/eNCSK5cGGIPmma11lXvVHa7uA88RMGBJPDyLPqbcV63qXAxyyT18bzXKHegkb0m5p+4ocsld\nuh7z3FNny2g3RnHE6dpwij7luc/NLR7T65drbBnpbqEJW8YFS0ObxCf3cJ5cmW+vhAuNK+OsrlDx\nh23C93YQUMgd+eRexZIBbOrkoUPyw0Gc1xvzyx1iyj1F7qtXdy8MjuRIeCQL96CR2Nnz4WIgkXG4\nCLgvTBXlHtaregeQIvcwI0dry4T2ihSoldQ5d02yZdxiEZJ+itw5K0lry6SsICmgygVe3VihfeM+\nn/4+CpfTHo7NzT1my7h6qXajo4sB2+PHi3IfaJx5JrB7t77+Y49VI/fly60ClVIYp6ZsuqREsD6k\n3a4acufapna1SrZM08pdWmS0yt1X5pzNFJJ76oz5mDIPN8gA6U1Mki3jB2pzbBl3TbpLkOwfZ3WE\nWTHcQWpaW8a9N9wcuVRIKZXQn4NT6qG9NDJi5xTeNXC2DEfSnC3j6nF9heTu2p04Uch9oJHruVdV\n7kDcdz9wQP/AbemcGi25h21TufFVlLuG3MfGFhWm65NbMKood4646yp398Wem1u85i9EWs+dU+5u\nzJgtw5G4I0TuDoKbf0hcfj+SXVTHlvEXAk1gM7x7CIncjX/yZLd94++a5l6Tb7n44/o+PLcwSl79\n8ePFlhlo9MqWAZoj9zrKnVsYUl4/57nnKPfjx3kyJupU5RK5V/HcNeSe8txjZ9Bz17R57uG1cEwp\nkw8OIHAAABkaSURBVEYiay6wK5F+TIlLRybUsWUkz31urvO1csod4Mk9LCOybY4diweD3SIQpkKG\nyl3y7/1FYfnyYssMPM44A3jkEf2Jbo8+Wp3cY0HVJpT74cPVbJkUuecq99DGSan8FLlXUe5camdd\n5Q4skrt0TfLcY3aPb8uE1/xFQbKJQhL3bRYtWXPlqcBsji3DnUPjz09S7qFKByy5Hz/evaAdPaqz\nZWKeuwso+wuFy8IqtsyQYXzcKuq9e3X1H37Yqv0qGATlztkyGnKv47lPT8spmv4Rxilyd8Forece\nI3dj+DuKGAn717kjj/1UzJBYY5aRb8ucONF5LUbubiGSUjLDcslm8cu1i0GYZlgloMrl1HPKPSTy\n0dFOIndlKeUu2SvhkQTLl9vPrPvdLdph6mdR7kOA884D7r9fV/fhh63ar4K2lXsvbZmcVMhUTnxK\nuY+MdBNgFc/dV+WOmHxFGNbhCNzNI9eW8RceiaTdvLXK3bWTPPew3NWXFLo2oMrdvVS1ZTTK/ejR\n7rug6em4cs9JhQzrrVjR3dfJk8VzH0qcf76e3Pfsqa7cN260nj2HXOXeb1smRtjhYhBT7v5CELNv\n/EVAGjsk93ChcGlss7Pyrtk6tkwsoBpbeGK2jFsUjOlebGLKfXZWrh+zZTTK3R/XzZVTt34fXJ47\nR+7hArNihf28hnbU4cM8uVdNhQz99NCWca/Nb1dsmSFAr5T75s3yw0FyyF06ykDTh6TcY4uC5Lk3\nZcuklDvQSY5Hj/L9pQKqfh0pyBsjWv966oCzkCj9ucXy50Nbxr1ut8EoTKHklLvvuWuVu3tNmmwZ\nf1z3Gjlf2r02Vx7uFNUo97GxbiJ3hB+Se2jLhHP3VXrKlvGVOxdP4GwZ97cfBBRyX4CW3OfnrTf/\ntKdVGyd2fnwOuW/YADzxRLU+ONWfOkmybrZMSuWnPHegcxGYnk4rd+msHFcnJFEHn2ilbJlQtfrX\npB2qLu3Tef2ccnfqPFTubr5S8Ffrufvkzi0SIbG6uwaNcpdsGW7MHOU+NtZN5FxZ6MNzyt3NxVfg\ny5Z1q3LO4gnTI4stMyTQkvtjj9mdptI2/RSaUu5r19oPX6gUtMp9erozO0jjuR87thjQnJ+340tq\nvxfKXSL3WEDV7yem3E+c6L59D9tzZBt7LUTywuAeEuHsolBta8hd8uI5cg9fl3tNHLEeP55ny0ix\nhhMnFt8PX7n7JBuqeddvFeXuDgE7fryb3EOSdvaK27PAKfcSUB1SaMm9jt8OxMn9scfsw0M0IOKP\nENaQ+7Jl9gvnk2+K3Jcts1+MUDmHwUiHHHKvoty1nnsV5R4SY7hb1s2Xs2VS8QN/YeF2xjoS5GwZ\nidwdKftzce9VeOfh+grJWlLoMeUexh0cOR892n1XEi6mHJFzKYiu/dSUjtxTGTRu3qF3Ho7JKXeu\nne+5uzjHoKCQ+wLOOMMSY5juF2L37vbIfe9ee10LzprRnla5dq39wjhoDizz7Rx3VIKEqtkysVMt\nq3jusbRKSbk7a0VaaNxrSyl3zraSlHbsWozcfVUsteGUuxTslco1yt1tJDp8uJvcw0VL8tc50uYC\nqpwtw5E7p8A55Z4id065O1umKPcBx8gI8IxnAL/9bbzefffZzJqqOO00q9C5DVP79uWR+8aNPLlr\nrJ1wYdCQu5/GmSL3MACrtWUOH5b7bdJzP3EivkCcOCEvNI7cuTlolTt318Cp4bCNtCBolburHy5c\nMUV//Ljso4eLh7NQOHL371Zii8nhw90LidaWkZS7m7vkuWuVu78ouDru/R1Kciei1xPRXUR0NxF9\nnLn+SiI6SES3LPz7y+an2j4uugjYvj1e59577SJQFStWWBINSdl9EXMeur1hg32Wq8PsrCWcGOk6\nbNzY2TZGqg7r1y/O+9CheP3webFTU/Li4QdrY4uMT5zSnUDYFxcTcMQi7QnwyTtF7mH7lHKP3TU4\nVRle89U5dz49p9zdPCTlHiN3LsNHyu7h8uunprptpXCO/uITkntowXAqnfPhQyL3y7iAqk/4R45U\nU+5+rv3QkTsRjQD4AoDfA/BcAO8komcxVf/ZGHPJwr9PNzzPnuB5zwNuvz1e57776pE7wGfM7Ntn\n/XbNiZAOofo+eNCS6ohiyQ7Jff/+tJ2To9w3bOiMBxw6ZGMEHHyLKLYIrF27aAtJ5O7XkRaKFLm7\nfQDSU61i5J9agFLKPea5c5k7uTaLqx/OzffuOeUe3uVIi0rMlvGVu3uYC3fHMTXVnedeJaDqynyS\ndndHvgJ3J7WG7fwdqpzn7voeWnIH8GIA9xhjdhpjZgB8HcDlTL0MWhpMPO95OuVex5YBbBpl+MzW\nXL8d6N4QlZNtw5H7xo3xNjnk7lS+S/ubn5cf4efOuQfiyt2/G5DI3Y8LSH05dS+Ru9sHoFHu4XVf\nkQJ8pg2nzv1+q6ZCcsqds2VOnuy+q3DKdHq6W9FLiwE3rmTLhEFk99hGrXLXqHnJlgnJPbRlpAWF\nS4VcUsodwJkAdnm/714oC/FSItpGRDcQ0XMamV2PcdFFceU+MwPs2gWcc069cc4+G9i5s7Ms128H\n7CLxyCOdfZx+uq7tqad2LgwacvfVeMqWWbnS3oUcO7ao2qW7klNO6SR3qV+n8I8ds3cnXDB0zRpb\nxxg5d9/1EyN3p9xzyX35cvs6p6b4xcdlHHHk7u4YuLTGXHKXAqouHfPgwe40zfFxuyD7r8lX7n59\nKT4wNmb7Du88jh3rJlRH7j6pSkTL7VDVkHvYH5cKyfn80iamMBA77OSuwa8BnGWMuRjWwvl2Q/32\nFGefbT+Y0tkvDzxgs2qq5rg7nHMO8OCDnWU7d9rxc3DGGZ13AHv26HfO+srdZYZoPHf33miOPXb1\nDx6MxxLWrVvcbZtS7lNTtm7M4jl8ePF22fdfw37aIHfAEsr+/XymjdtjwC1iExN2XmG/o6N2MXvi\nie73xhF/GF9w5MnN0REwZxnt399tvzhy1yj3iQnbR6jcp6bs/26Bdxu6wgPBJOXO2TKSVeOT9OrV\nVjiE2TKhRRQuKFxA1e2f8FMf/YCqv/ltEMB89LuwB8BZ3u9bFsqehDHmiPfz94jo74hogzGmaw/l\nFVdc8eTPW7duxdatWzOn3B5GRoAXvhD41a+A172u+/pvfgO84AX1xznnHOAHP+gse+AB4Nxz8/oJ\nz6HPORZh40bgzjvtz088YYk45fevX2/vXAAduTulf/SoTMZApy0T89zXrbN3KjFyd8o9tuPW2Tsx\nz/3w4Xi2zL59ljyl648/LscEpqZkct+3z5JNuH9gfNxmWYWvaWLCfnbCRdGR++HDwFlndbYZG7N/\nl3Du4+N23r7t6Cv3kNynpzuftwrYPh9/3O4b8eseOtRJ+EQ8aa9c2UnGrn1Yj7NlJiZsTMwfZ9Uq\na3n6mTEjI7atbxGFNiOn3MNAcbjAcEd0VMXk5CQmJydr9aEh918CeAYRnQ3gEQDvAPBOvwIRbTbG\n7Fv4+cUAiCN2oJPcBxGXXQbcdBNP7tu2ARdfXH8MTrk/8ADw8pfn9cORuzYH31fujz6q2zy1fr1d\n4AD7BX72s9P1DxywJBNT7r4t88QTcmBXo9wdMcfuANats/0cPcq/bo1yn55Okzun3NeutddmZ7tj\nEGvW2MWLm/fEhP37htf8xcK/5kiZyxhydxanndZZPjZmy5///M5+JM/98OHuu9iJCfv6QuV+6BC/\nJ+DgwW47aefOzvdVG1CdmLALoP++r15tX1M490OHFus55R5aPGFAdWqq83M8MWE/r22Qeyh8r7zy\nyuw+kraMMWYOwAcA3AjgDgBfN8bcSUTvJaJ/v1DtbUR0OxHdCuBvAbw9eyYDgpe9zJI7h23bmlHu\n551nA7M+qij3M86w5O6OBMhR7v5zY7W7bv3US81zZF1Q9bHHuonEh7/TNlbXKe6DB+XAsU920tEI\nrp/Dh3lydgFVSdm7RUsi/1NOAR56iF/Q1q617zv3rFxH4Jw95voMr7m7npDcV6ywC8jBg90Lwimn\n2EWES9MMbRm36/LIkU6CXL26m0jdawhtGaeMw8XMLTL+eK6MW0hStszq1Vao+G1XrbLvD3fX4dsy\nJ0921hkf73zNK1bYHbX+63WfDbc4cQ+e7yc0yh3GmO8DeGZQ9t+9n78I4IvNTq0/eNnLgHe/uzuK\nPz8P/PznwJe/XH+MLVus6nPWhjHVyH1iYlFtnXpqnufu3z1oyf3MMxc9fi25Hzhg5xc7aG3TJmtH\nHD1q32dps5PbAHbggKzcXXpobH5OuU9N8XWctSNZT27zGBFPxBs2APfcw9+BrF0L3H03325iwlpl\nErnv2tWdhrt2rSW0Zcu6A6erV1tLIiR3Z2+FxLxmja3vv//uMYihzeTmE/btbJmQ3A8e7F60x8ft\nguUvoC6oG5LxgQPd5BtaS64f/3W5NmF/fj03V78v97NrF7YBFl+7q+viKYOCskM1wMaNwHOeA/zk\nJ53l27fbD6c2GyUGos4NU/ffb79w2jRGH898JnDXXYv9aDN5Tj3V3m4fOqQn9y1bFtX+rl3A058e\nr++OWti7N/6+TUzYoOE999j3WPL+Tz/d9uViBBxWr7ZEd889cvaRC+BKZ/ls2mSvSXcRztKS7KyN\nGy2Bc9lH69YtKvcQExP2bxEjd86W2bOHt3I2brQWh1a5b9hgxwgX140b7Zz9+u7JZVwMwFfFru4T\nT/DKfXa2W7kDnWS8Zo1Vzf5Yjsj9hYEjd9e3X+YHRKUxXbuwfWgDhXULuQ84Lr8c+M53OstuvBF4\n1auaG+Oiixb961tvBS65pFo/z30ucMcd9nbw8cf1GTdEdiHYudPeNWjabdpkF4NDhyy5pe4SXMpn\nitwBq+xvuy1u3zhy37XLLjSxerfdJpO7SyGVyNmRyYMP8vNxdwdS+w0brO3Gkbtvy0jXOKKWbJm1\na+3dFGcfnXoqH4R1i1t497Nxo81g4ch9fr6b3IHucV1bv2939xOO5/oLUy/DMjd//3W494Ej99CW\nCcucjelEBDdm2M5tDPSPDQnJfWxs8aEfg4BC7gze+lbguus605q+8Q3gbW9rbozLLlu8O7jlFpul\nUwWO3O+6C7jgAvmURg7nnmvV/o4d9m4lhZERq9Z/8hOr9FNjnX22JcidO+NkDFjC/fnP43cDq1db\nhX/HHfF6mzfHyd3dgUjkTGQXiNtvl5X7o49aW4Aj8A0bbNYGZ8usX2+vcbaSe1C7pNxPnuSJGuAX\nBEeqnHJ38wxfF9BN2K7cv1tyP4dxBdfWf31uHG5hCstdf/5Y7rr/OjjCd2P7r8uRc2pPhl8X6LZl\nHPzPfGjLOCtsUNR7IXcGF15olfVXv2p//+UvrWJsMmvzda8Dfvxjq5RuvLF63xdfbOd3003ApZfm\ntX3BC2zbu+5KZ774bb76VeBZ3AEUAVzg+Pbb7e7fGC68EPj2t9P9nnUWMDkZJ/czzgBuvln2+Tdt\nsuQ8PS1n1GzezGeUAJYMZmbsYsfl0TtS5YjfpQhydz3uNXHXHNmFr8mRoRQb4ObhiDcsd6QYLorO\nigh3rgIyYfvk7N6j0G4L7RGAf+/cfP0+uTJH7n6ZI95YfMh9Bvwx3XsRvkf+39uN5y9whdyHAJ/+\nNPDJTwI/+xnwoQ8Bn/gE/0Wuik2bLFF++MP2tvqyy6r18/KXW+X9+c8Dr31tftvPfMbmNWv9/n/1\nr+xdzMtelq77rGdZ7/vAgXSq5SWXWNWaIvdLLrFfngsukOu86EX21lvqy91iE8n+viO4MEfctQMW\njxgI4QicO6bCBUS5Oxk3FrdwOdssbBceb+DDWQihzeIsstCKc4QZWmjulE0f7j0I797c3LnXEHru\n3MmojoT9z4t7zX5cyP3slzlC9hdkR9z+3cjcXOeY/nnyDu7vHy507lA4v29/wV2zhn/8ZV9gjOnZ\nPzvc8ODqq405+2xj3v9+Y+bmmu//F78w5oILjPnGN+r18zd/Y8xllxlz8mReu5kZY179amO+/GV9\nm4cfNub884256y5d/Q98wJiPfzxd7/777XvxyCPxet/6ljGvelW8zm9+Y8zv/E78/fjoR435r/9V\nvn7ttcb8/u/L1z/5SfuPw+OPG7N2rTEHDvDX3/hGY265pbt8ft6YF7zAmN/+tvvanXcac955tk6I\nP/gDY665prv8m9805jWv6S6//XZj3vSm7vJf/MKOEeLv/96Yf/NvussvvdSYr3+9s2zXLtt3OM+P\nfcyYH/2os+yqq4z53d/tLLvnHmNGRzvLpqaM+aM/Mub48cWy6Wlj3vUuY44eXSzbv98YoHPsn/3M\nvqc+rrjCmH/7bzvLPv5xY7ZvX/x9zx5j/uzPOuv8yZ8Y8z/+x+LvTzxhzFve0lnnAx8w5n/9L9M4\nFrgzi2/JuOhCD0BEppfjFRQUFCwFEBGMMVmHMxZbpqCgoGAJopB7QUFBwRJEIfeCgoKCJYhC7gUF\nBQVLEIXcCwoKCpYgCrkXFBQULEEUci8oKChYgijkXlBQULAEUci9oKCgYAmikHtBQUHBEkQh94KC\ngoIliELuBQUFBUsQhdwLCgoKliAKuRcUFBQsQRRyLygoKFiCKOReUFBQsARRyL2goKBgCaKQe0FB\nQcESRCH3goKCgiWIQu4FBQUFSxCF3AsKCgqWIFTkTkSvJ6K7iOhuIvq4UOfzRHQPEW0jooubnWZB\nQUFBQQ6S5E5EIwC+AOD3ADwXwDuJ6FlBnTcAON8YcwGA9wL4Ugtz7RkmJyf7PQUVyjybxTDMcxjm\nCJR5DgI0yv3FAO4xxuw0xswA+DqAy4M6lwP4CgAYY24GsI6INjc60x5iWP7gZZ7NYhjmOQxzBMo8\nBwEacj8TwC7v990LZbE6e5g6BQUFBQU9QgmoFhQUFCxBkDEmXoHoUgBXGGNev/D7nwMwxpi/8up8\nCcCPjTH/uPD7XQBeaYzZF/QVH6ygoKCggIUxhnLqjyrq/BLAM4jobACPAHgHgHcGda4H8H4A/7iw\nGBwMib3K5AoKCgoKqiFJ7saYOSL6AIAbYW2cLxtj7iSi99rL5ipjzHeJ6I1EdC+AaQDvaXfaBQUF\nBQUxJG2ZgoKCgoLhQ88CqpqNUP0GEW0hoh8R0R1EtJ2IPtjvOUkgohEiuoWIru/3XCQQ0Toi+iYR\n3bnwnr6k33PiQER/sTC/24joq0S0ot9zAgAi+jIR7SOi27yy9UR0IxH9loj+NxGt6+ccF+bEzfOv\nF/7u24joW0S0tp9zXJhT1zy9a/+JiOaJaEM/5hbMhZ0nEf3Hhfd0OxF9NtVPT8hdsxFqQDAL4CPG\nmOcCeCmA9w/oPAHgQwB29HsSCXwOwHeNMc8G8AIAd/Z5Pl1YiCX9KYAXGmOeD2tVvqO/s3oS18B+\nZ3z8OYAfGGOeCeBHAP6i57PqBjfPGwE81xhzMYB7MLjzBBFtAfA6ADt7PiMeXfMkoq0A3gTgImPM\nRQD+71QnvVLumo1QfYcxZq8xZtvCz0dgyWjg8vUXPoxvBPD/9XsuEhaU2iuMMdcAgDFm1hgz1edp\ncZgCcBLAaiIaBbAKwMP9nZKFMeanAA4ExZcD+IeFn/8BwFt6OikG3DyNMT8wxswv/PovALb0fGIB\nhPcTAP5fAB/t8XRECPN8H4DPGmNmF+o8nuqnV+Su2Qg1UCCicwBcDODm/s6EhfswDnLA5FwAjxPR\nNQv20VVEtLLfkwphjDkA4P8B8BDs5ruDxpgf9HdWUWxymWjGmL0ANvV5Phr8XwC+1+9JcCCiNwPY\nZYzZ3u+5JHAhgH9NRP9CRD8mohelGpRNTAyIaALAdQA+tKDgBwZE9PsA9i3cYdDCv0HEKIBLAHzR\nGHMJgKOwlsJAgYjOA/BhAGcDOAPABBG9q7+zysIgL/Agok8CmDHGfK3fcwmxIDY+AeBTfnGfppPC\nKID1xphLAXwMwDdSDXpF7nsAnOX9vmWhbOCwcGt+HYBrjTHf6fd8GFwG4M1EdD+A/wngVUT0lT7P\nicNuWEX0q4Xfr4Ml+0HDiwDcZIx5whgzB+CfALysz3OKYZ87t4mITgfwaJ/nI4KI/h2sfTioi+X5\nAM4B8BsiegCWl35NRIN4N7QL9rMJY8wvAcwT0cZYg16R+5MboRYyEd4Bu/FpEHE1gB3GmM/1eyIc\njDGfMMacZYw5D/Z9/JEx5o/7Pa8QC9bBLiK6cKHoNRjMAPBvAVxKRONERLDzHKTAb3h3dj2Af7fw\n858AGBQB0jFPIno9rHX4ZmPMib7NqhtPztMYc7sx5nRjzHnGmHNhBckLjTGDsGCGf/dvA3g1ACx8\np5YbY/bHOugJuS8oIrcR6g4AXzfGDNIXCABARJcBeDeAVxPRrQte8ev7Pa8hxgcBfJWItsFmy3ym\nz/PpgjHmN7Anmv4awG9gv1BX9XVSCyCirwH4GYALieghInoPgM8CeB0R/RZ2IUqmxLUNYZ7/DcAE\ngP9/4Xv0d32dJMR5+jAYAFtGmOfVAM4jou0AvgYgKejKJqaCgoKCJYgSUC0oKChYgijkXlBQULAE\nUci9oKCgYAmikHtBQUHBEkQh94KCgoIliELuBQUFBUsQhdwLCgoKliAKuRcUFBQsQfwfmlMJjLs6\nqBkAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fe80fec9128>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import numpy\n",
    "plot(numpy.arange(0, 15, .01), \n",
    "     [pyfunc(x) for x in numpy.arange(0, 15, .01)])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we will compute the derivative of our function. Normally, in regular Python, we would have to solve for the derivative. However, because we have described the function symbolically, Theano can solve for the derivative via the symbols.\n",
    "\n",
    "To do this, we will use T.grad() with respect to (wrt) x:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 229,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "fp = T.grad(fx, wrt=x)\n",
    "fprime = function([x], fp)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 230,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(10.394080663811636)"
      ]
     },
     "execution_count": 230,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fprime(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's plot the derivative of the function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 231,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x7fe7e5f1f390>]"
      ]
     },
     "execution_count": 231,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXwAAAD7CAYAAABpJS8eAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJztfXu0pUV156+6+3Y3dDd0N0/tljcoggrEgYwY0xEx4gPM\nmGA0E5+TZHQSzJiVGAwzNIkZ0UyWyyQ6S0dU0ATjK0oUBJFcDGQEA/JqHjY00EC/pB+Xft5nzR91\nt2efOntX1fed7/Q5t+/+rdXr3lNV33fq3r7nV7/vt3ftct57GAwGg+HAx5x+T8BgMBgM+wdG+AaD\nwTBLYIRvMBgMswRG+AaDwTBLYIRvMBgMswRG+AaDwTBLMK/fE3DOWV6owWAw1ID33lUZPxAK33s/\n8P8uv/zyvs/B5mnznMnznAlznEnzrIOBIHyDwWAw9B5G+AaDwTBLYIRfiFWrVvV7CkWweTYLm2dz\nmAlzBGbOPOvA1fWCGpuAc77fczAYDIaZBucc/EwM2hoMBoOh9zDCNxgMhlkCI3yDwWCYJTDCNxgM\nhlkCI3yDwWCYJTDCNxgMhlkCI3yDwWCYJTDCNxgMhlkCI3yDwWBoGDfeCHzxi/2eRSdsp63BYDB0\nie9+Fzj/fGD+/PD6hBOAxx8HekltttPWYDAY+oA3vhH4539uvR4a6t9cUjDCNxgMhgq47TbgG9/o\nbJ83T/5+kNDYtJxzcwD8O4CnvfcXOueWAfhHAMcCeALAxd77kabez2AwGPqBd70LeOyxTruGq/rZ\noPA/AOBB9vpPAdzsvX8hgFsAXNrgexkMBkPjePJJYNeu1us9ewAXueQLF8rXzhrCd86tBPB6AJ9j\nzRcBuHr6+6sBvLmJ9zIYDIZe4bjjgPe/v/V6fDx8HRtrtcWET0qfk/ygWjpNKfxPAPhjAPwh5yjv\n/WYA8N5vAnBkQ+9lMBgMPcO2ba3vJyfD1xFmRh90UPt4Wgy4xTOohN/1tJxzbwCw2Xt/j3NuVWKo\nmqC0evXqn3+/atWqA/rEGYPBMBj4y78MqZOf+1x7+xwmgznhH3FE+H7BgvbxRPj0NAD0Jh1zeHgY\nw8PDXd2jiXXoXAAXOudeD+AgAEucc18CsMk5d5T3frNz7mgAW7QbcMI3GAyGpvH008D11wO/+7ut\nti9+EXj00U7C5579xET4um+ffm9aFGgsAExNdTVdEbEYvuKKKyrfo2tLx3v/Ye/9Md77EwD8JoBb\nvPe/DeCfAbxretg7AXy72/cyGAyGHHbuBF75yva2z34W+L3fa2877DD5eonwOYHHZE5jOOEP6l7S\nXubhXwngfOfcIwDOm35tMBgMjWL9+vbXjz0G3H57e9vSpZ3XxV48gRM+qXf6CnSS+f5S+E2g0dCC\n9/5WALdOf78NwGuavL/BYDBwrFkDnH56OwlzL51wyCGdbZoKr6rwJcKfjQrfYDAYGsXjj7e/3r27\nc4xE+HMEpishZSJzTvIlCt8I32AwGArhfSdp3nZbKEoWj4sxOqr3cWi2C7dviMR5m6bw+UIzqJaO\nEb7BYBg4rF4NLF7c3rZhQ+c4IlZO7nv3hq910yQlD79E4adsn0GBEb7BYBg43HVXKGvAMSJU4iJy\nJ1UPyLtjJQJu2sPn9zNLx2AwGAScdx7w0Y+2t82d2zmOe+QEInyeJ08EzAlfImBNhddV+LzdFL7B\nYJjVmJgAXvvazvZbbgFuuqm9TSL8uIgZ0FL2RPz0PoCs8EtUOA/wVvHwU4vCoMAI32Aw7Bfs2QN8\n//syGcaboEoJXyJkauM2TxVSzlk6MczSMRgMhghSNgshJngpjZKImJOpRPjSZilpXAnhm6VjMBgM\nGbzpTcAll7S3EdFLdWliEk15+NzLlxQ4teUWhqYtHSN8g8FwQGNsDLjuus7273wnFCuLxwJlhC8p\nfCJf/oSQsnSkRaDE0pGqZVZNyzRLx2AwzGh861udRPblLwMXXSSPjz33KgqfEzghRfhNKvwm0jJN\n4RsMhhmLiQng136t/fg/IFSm1BATahWFLxGmRPj0veThS4tAivDpdd3iaRa0NRgMMw7PPtvZ9txz\n4Wu8GaoKqij8FOHnrJq6lo6UullF4ZulYzAYZhyOOAK48872th07wlepWFkpJDVO6AXh5ywdra59\n1YXCLB2DwTDw4Ge3EkjBx0FTUvgx4Uu58RrI0pEIP0aK8HNELpF0FWsmd515+AaDYUZh376w2Wnz\n5vZ22rHKd6ny1zHhE8mVEBwpfGlsnIaZCtrmFHhdDz91XVWFb5aOwWDY7xgZCWe3xti4MXyNPXna\nncp3qQLyQd38tbSZKkbK0omfKKpaOpLCz3nxVRR+VQ/fFL7BYNjv+PzngXe/u7OdCFwj9jiwSmQd\nFzCrQvgpS6dJhV+6MMSEX/c6U/gGg2G/4t3vBm64obNdyrgBdGLXFL6mzqVSxISY9FIplDG68fBL\nLZ0SpX6glVZo9Exbg8HQH3zxi8D27cAFF7S3SztWAZ3wNeWfU/gS4cfvLSl8icSB3mXpVLVmTOEb\nDIa+4LHHOssVcEj57VJNGiDv1WsLgUb4EsHF7y0pfC2Q26s8/JQKTy0UqevMwzcYDI3jgx8E3vAG\nvZ/XhCcQ6ZbueqUFICYszdKh+5Rk3kjkrpUfbiJoW5rNE7+WxpQQ/kywdIzwDYYBgxYAXbo0fZ3k\nh0unP/HXmsLXvHpN4ZcEYqV7awpfup/0NJEi8iaUepXrzNIxGAyVcNNNwPz5cp/kk3NI9o1mxWjE\nnmvXCL+Kws+pce1+TSv8ukFbK61gMBgawaZNel+sxmNIAVq6ppTwq1o63RK+dn3TdXOqePhNKHyz\ndAwGAwDg298GXvUquS/liedUo0T4OcUev09dS0dSxXHZhVSWjrbA1Mmvl0i6xJ8vVfjadUb4BoOh\nAzffDPzrv8p9VG5YyrjJQbJ0NIVP7aULgUb40ngtc6eKwk/ZP6UefmmqZorwSxYPifBL6gX1A0b4\nBkMPsHAhcNVVcp+WKgm06s3XKUPchIefWwg05V9C4tLikFP4dbz5ujZP3QCwpWUaDLMco6PAD34g\n92mboYCWspdSLKXKjbn75jz8Xlg6VVR7bmxTlk7J00Fp8bR4ESjx+QcFRvgGQw2sWwfce296zDxl\nH3tK4acIX9ssRUST8vCrZul0E7Tt1qbJjY0JeWgob/OUEHdVK6jE0hm0bB0jfIOhBt73PuCMM9Jj\nhobkdumoPoKmyHmfVrFSUpVNWzpVPPySp4c6WTox+Q4Ndefhc+KeO7eTuJ3rvL9zetDWe3kOgwAj\nfINBwPh42kePz3atAlLvEuGTwpdsG42MtYWA9zWl8Eva6yh8bWyJcp8/v8y+mTs3/XSgPS1obZoV\nNDUVFoQ5c0zhGwwzAu94B3DiiXp/KotG87vja+sq/JikU/fTlHmdfPu5c5uxdDgJjo93KmitPUXa\ncdu8eemFwfvQz8dNTnZeNznZTvhTU+F7/iQQPxnQffhTwKDACN9gEHDrrelNUKmj/UjBSz48b5dI\nnci7CuFr7byvisKfM0cm4IULywOx0uIg3XtiAliwQL5v3K6Re6zwY5KWriWS5iq87nXSk8LcuUb4\nBsPA4IEH0h/GRYvS16cCr0TomiWUIvwUeWuLQcoGIpItJXwidomsS4lZakuNnT9fHitZNVKbtgik\nFgZ6CuBPEdriMW+erN75dZzwaVE4IAnfObfSOXeLc26Nc+5+59wl0+3LnHM3Oececc7d6Jw7tPvp\nGgzdY2oKeMlLgDvu6O4eGlIePe+vo/Cd0y0daU6jo2HxiucyNhbITSL2gw6SFwKJxMfGOpV/aqyk\n2qsofInIJcLPtXHCj5V67rqYzOnJIH4KOCAJH8AEgA96708D8B8B/Dfn3IsA/CmAm733LwRwC4BL\nG3gvg6Fr0EHcW7boY3LZFak68ClCB8psG63v4IOrWTpE4BL5au2addOUwo89/JjE6efQfPiSoG0u\n+ErEnbJmpDZu6WgKn14fkEFb7/0m7/0909/vAvAQgJUALgJw9fSwqwG8udv3MhhK8Fd/BfzsZ3o/\nlS/Ytk0fk/ugpkg7R/i5LB2J1AGd8Pftk9U6XSMpdq09ZelI7VUJP15MJIU/Pi4TJrVLQdW4TVpE\nYhWes2b4dTGZSwpf8vAP6LRM59xxAM4A8CMAR3nvNwNhUQBwZJPvZTBIGB8H/uRPgFtu0cdQSuWO\nHfqY3AeV/HmpZHEp4WsqfvHiagqfFgktD18i9lS7pvBT7XHQttTSkRQ+J/wSNV/F1489/Fjha/eP\nrSDpyUBaTA44hU9wzi0G8HUAH5hW+vGPOmA/uuFAxObN4at2eDdQVqCM51RLSKn0vXuDb56ydBYs\n0D186VrvdZLWFoJUX8qrL1XydJ9SDz/ly8ckG6tvaq+buZPy8GOFn8vVj6+LA7tSfGBQ0Mgh5s65\neQhk/yXv/benmzc7547y3m92zh0NQHVMV69e/fPvV61ahVWrVjUxLcMBhptvBoaHgY98RB9DvvzI\niD6GCD9VX54UPJFUjL17A6lohL9kSVrhH3JIWuFLpD40FIhEU/gpD18jfM3b1xaCXnn4/PeoWTpV\nPPz4Z5MUvubh86c27clgbCydltkLwh8eHsbw8HBX92iE8AF8HsCD3vtPsrbrALwLwMcAvBPAt4Xr\nALQTvsGg4eqrgS9/OU34zz0XvqZ2wnZL+N6H/sMP1wn/kEPSWTrLl+sK/6ij5I1SCxbIhE+ZONLP\nkwvOlhJ7E2mZmqWzYEH73FOWTkkKpuTrV1H4uSeDXP5+r/LwYzF8xRVXVL5HE2mZ5wL4LQCvds79\nxDl3t3PudQhEf75z7hEA5wG4stv3MsxuHHxwfgyReYrwqU8j/KmpQLwaaY+NBbJYuLC+wtf6ibwl\nwl+4MBCJlIcvKfypqXQQNuXhl1o63ebhS4RNhC9ZOnX8eqmNbBiJuOvk70se/iBm6XSt8L33twPQ\ntqG8ptv7Gw58/PCHwBFHAKeemh5Hu1vpQyehhPBzCn/PnkCGmmWzZ08g2KGhNOHT0waH93puPKB7\n+KTwpR2sGuGTXSJdMzoKHHqoTMqLFpUFZ71v+eqlQdsSvz+VpVMlLbMphc/nLAVtUx7+AZ2lYzDU\nwS//MnDxxflxRNRayQIas2RJK9deG6NZIEC4dtGiNKEfdFCe8DXLZsGCQFJSYJYWA4mgNcKna2Jy\nGR3VCb+qwtfIet68znz5phR+bOmUlluIlTpVr8z57KV5+LmFYpCDtkb4hoFAiqAJ27eHr7kqlocd\nJqdL8jGHH16f8OkJoA7h02Ixb15n/8REIImFC5tR+OSZxwXHqK9bS6cKMfP2WLVr940zYYB8UTRq\nK9ksJQVtc1ZQyULRKw+/CRjhG3qCiYlqf+ypU6AIRPg5hX/44WnC37kzLArdKPwSS0dT+AsXyoRP\nfZJPn1P42g7c+fPD77ZqWmapV1+F8LXSCimFT39DVTJ3aFNZrhxClbTMnKUz6/LwDQaOk09OZ9MQ\nUsQco5Twcwo/tyg0ZemkrpUIP5eJk1P4GsGmLJ1usnfqKvwqHj61V8nckQhf2iylbaAqVfhm6RgM\nCH/kTzxRVpyMSJz8+RR27ACOPjpt6eTUO43p1tLJKXwtzz5F+Cn1X8fSyXn4GrGXLgRUgC1+gqha\nWqEkS6d09y1tUOPWkabm66ZlagrfCN8wK1GSC0/Ytg1YuTJP+N6HxWHFiu4Vfs7D37Ond0Hb1LWc\n1Ovk4ac8/FIbqJcefmlpBYlUpUWAvvIYxdhYa4NaTuHXKZ5WkpYp3XtQYIRvKMKdd8o7OSVs3Bi+\npgqYEbZvD4Q/Opq+/9694QO0dGmZh59T+CUe/rx53aVl1vXwc5aOloefytKpuqO2CuHH908p/JJa\nODlLJ2XL0O8pp+arePi5oG1JzfxBgRG+IYstW4BzzgGuu65s/KZNwcPfujU/dtu2QL6LFqWfCLZv\nD2R/8MFllk6Jh58ifCL0qiqd+utk6dDmqqYsHVLUqaBtSa480IyHXzUPP7Z0iNxTRM5tJv50IKn5\nnDVD43KWjnn4hgMK998fvt57b9n4jRuBk04qS7Xctg1YtizUj0nZOjt2hHEHHVRm6aQUfhNpmaUK\nP/7A5zz8nKWTsmckgpU8fPK5U8HZkhOv+IISE/bCha33Alr3KymNkMrSiRV5TOT0e4qfDubP71xA\n4hLGVdMy+dOCEb7hgMGDD4avZNXksGlTOAB8z578I+327aGuzJIlacLfvr2c8PdHWibtxNXKIx98\nsKys9+5Nq/gSS6fboC0RkvQ+kgr3XlbiPO1Ta6f7S6RL7XWzdLjapjbJ0pEWEFqs6px4lTspy/Lw\nDQODUh+e4/HHgbPPLrNogLAwrFgRiCNFzkBQ+MuXB4JNWTVVLZ2ch798uV7crIm0TOqXVHxO4Ve1\ndEqCthIhS/eSFP7kZCDLeIHQLCNJ+VMwVSqXUGLpEKlKtgwfxxU+J/d4sZHGSUFbmkvOCtI8fCN8\nQ9/gfVDJn/pUtes2bAhnwKZqzHP87GfAkUcGmyaXqUOWTlwxMQYp/IUL03Xscwrf+0Doy5bVJ/wS\nS0cjdd4nZenU2XiVC9pKhCwFc7nVEy8QuYUjtnrids3vl2IJpQpfKlKmWTrxYqEFd5vMw7csHUNf\nsW5dIMTvfa/adRs3Ai99aTnhP/tsIN0SwidLJ0f45OFrJAsEAti3LzwJaIS/Z08gtAUL6hM+z+Kp\nGpgt8fCbDtrG12jKfHIyENnQUHu7RKLxfXJWj3YPWuTqeviawo8tnVKF382JVzOheFpT9fANMwB3\n3gm88IXBoqmCjRurKXxO+LnALVk6OeVOCp/8ZAm7doX3XLhQXzyouFpq4cjl4XPClyyyks1VExOy\nNZPz8LU+Kh09NdUqU6FZN1owl6dZlhB+lYVAWxwkhc/tn7ppmbGlU0XhSx5+SdDWPHzDQGHdOmDV\nKuCpp6pdt3Ej8OIXpw/95qii8KtYOkuXdp5IxEFkTptuJDLeuTPMK0X4tHDkCF+yXoAyha8p9ZIs\nnVR2j0S8pYSvWT0pYpeCtlI7LT6x6pUUPsU5cjtteZZOSdC2qofPq2yWBG3NwzcMFNatA846K/yB\nSrXaJezaFf6Ajzwy/NFrJMmxdWsInDZp6ZDC12rUA+FnOuSQ8EHTMmh27WopfImsaUyK8HftKrd0\n4utzhE8eftXyyAsXdlo0mseuZdeQ9SHdR7KAUu0pS4eToBQzIMLXiqfxey5cmM/D19rihSFOr+RV\nNqvk71tapqER3HVXIN41a+pd//jjwAknhAyaDRvKrtm4EXje88If76JFeYuGFhPKqCm1dKp4+JrC\nJ8IH9PvxpwBt4cjZPilLZ2qqRWJVFX6O1FMevtRHC0hMyESoksKXds7yhSgXnOXtJR4+/a44MVLq\nqmTpcILmP19J0DZW+NIGrZJNViWlFYzwDV3j//7fkAHzta/Vu37dukD4y5eX2zNE+EA1i2bu3DyJ\nU32cpUurWTo5hQ/o47ilo5Vw7sbSIfJ1rrqKr6vwaWdwTLza4kKEKrWTsu6Fh6/l7Fe1dOj/jH5f\npUHbnIdPVTZz5B4vAnGxNvPwDY3gxhtDyeEf/7j6tePjgbxf8IJAyFSlMocNG6oRPvn3QFkKJaU3\nlgZtSywdIE34S5a0CFmyZGhR0IKyKYVPm64AvT+n8GNSBHQVPzkZ7rloUSdRa+/FLZMSJV/Vw09Z\nOrGHLwVtucJPWTo5hR8HbbWFgV/LnzY0v17KDOLXSemigwIj/BmCrVuDen7rW4EHHqh+/fr1wPOf\nH/7wly8vJ/yNG8N1QJmlwwk/p9q3bg0kXjKWCL/U0kn570uWlI3RPPqUwieSBeR+8v8lwqf8fq1P\nuo7mMmdONSUvEbu2ENRR+KUevqbwNUsnJvz4/VOlFXILg0b4mqVTslAMWlqmEf4MwV13AWeeCRxz\nTCBhLeCo4fHHgeOPD99XUfhVLR0K2AJ51b51a/nisGNHsHRSCp/UO5C3dLQx3rcr/BThpzx6QLdf\nNMJP9WmEz3/mlKVTQuySdw7owVmtHr5k6eQ8/FJLR/Lwc5ZOTuHHlg5/z5i4eRE3/gQRH3Rulo6h\nK9x1F/ALvxA+YEceWR50JZB/D1S3dEjhV7V0ShQ+LQ6psePj4cNNgdRuFP7ISFg4AJmwx8ZaWT4l\nhK9ZNoBO6osX1yN8Sf1zwteCs5LVIxF7UwpfsnRSKZykjokcJUuHZyLFHn5Ouadq6aQUvmYP8YVi\ncjJ85Vk6kl00KDDC7xH27QOuuiqUFm4CRPhAUPlPPlnt+roKn3v4VS2dnMIvHUsBW9oF2o2HT08K\n2hhu+UiET6UZNEsnVtxVFb5m6WiB2dT7VfXwUwtBr7J0JHKU5ifZPCmFL1k6JQqfE76Uusn3E1Ap\nCikWYIQ/y/CHfwhcfjnwW7/VzP3uuSdYOgBw1FFlh4twrFvXIvwlS8pOowLaPfx+KXx+z26DtjnC\n55ZPyRNATMzdEj71xd4vKfxYrZdaOvtT4acsHSloGy8OND8+VpqzpPqlDV6pnbY5hc/JXfrZpPRO\nXqzNCH8W4IkngK9/Pajy++4LZNsNdu4MxHvKKeH14YfXI3yydKoQPlf4udLEQJhXqcIvJXzu9acs\nnZGRPOHT04I2hlIyAZnwiZSBvMKXrs8FbaVsG+ojwtcUfipLJxcUpfE5Dz++T+nGK2nzEynkeKxk\n1Ui+vqTwJatGU/hSG/fwtUVBU/ipDKBBgRF+D3DNNcDb3haU+GtfC9x8c3f3u+8+4PTTwwcXAI44\nohrhew+sXRsOJQHyh40Qdu8Of7yUSZMrTQy0B3lzCr/0aeDZZ1sLQ0rh0+lZQHcKP0XYnPCl/qYU\nft2gbUlevdauLQQa4fMFihObtDhpVgsFQGM1n7J0Sj38qgqf7pcqqxwvKLmnACP8AxzeA1/6EvDb\nvx1ev/KVwB13dHfPe+4Bzjij9frww8sLmQFhrHMtMiyxZoD2XbZAmcLfsCHs5AWqKXytFALNv0Th\nx+NyhJ9S4Fo/pYdSv2Tp0BNCTMBTU608/br+vqTw6f0kxa55+JInLxEtoNtJPCWUk/jUVJmHT4Qa\nz11S7tLTh7RAVa2lk9p4lVP4ZOnkxgwSjPAbxh13hP/8//AfwuuXvjQo9G4QE35Vhf/oo+GMWSLu\nUsJfvz4EiAklCp9n9SxYkA/a1rF0NIVflfClMXwRkgifv0fVoC3Pf69C+N7r5F36fprCL7F6yOaK\nFwhJyVObc+0LhESwPKNJsm+0tqpBW8lTj20X6SwAya6Jg7Spw1XM0pkF+PKXgf/8n1vkevrp4YjA\nOidNEbol/LVrA+ETSgmf+/5AXuHv2RP6SQGnyhQD7YtDTuGXWDpNEP6WLSHtFdAJ/4gjWv1VgrYj\nI8Chh8p9QLpEAmXDpDz8+Lrdu0OftFlLelrQ0jgplTQeLyl8bnlxwuaLQ9wGlKv50lo6cdZPPC62\nXTTbR1ooeCxCej8pc2dQYITfIMbGgH/8x0D4hCVLgpf/2GP17jk6Cjz0UKhHT6hq6cSEnzs/lsBT\nOYG8wqeMHlrscgr/mWda9k9plo5m6ezZEz5cvKxBTOaTk2GhSwV2N28O/190j5jweVA65+HHCwIV\nigM6yXNqqjW3mHCfe05fKFJBW8paklI56X1iApcWHLK5UlZPrPDjn1Eay8tQ5AK0pYuA5s3H94uf\nOLRSC7k8fC0t07J0ZgFuuAE49dR2kgRCsLQu4d91F/CiF7V8WqB6ls5DD4WDTwi9UvjPPNNS7ED+\nIJKJiZbaTil8TrKawue2DyCT+chIS6lqY7ZsSRN+ztLhgeO4nxO+pP7JNon7+FNJqcKfnGxX5tIi\nod1Ls3SqKnxOsNJYbXGQNl5pNk9K4Zc8CcRKPUXudF2s8FOll43wD2Bccw3wjnd0th9/fPVTpgj/\n+q8h8MtRpRYO0GkJLVoUPsS5P8aqCj9eIFIKn9Q9fxrQFocnngCOOy58ryl8TsQ0LkXmgEzomzd3\nZ+nEC4Km8OM+HgxOEb6kviWFz1V5fD9S/loAOEX4/D4lCl+ydKSxfOcqHStYxdJJ+edSXr8WfKW/\nxxIyj6/TFoVBghF+Q9iwAfiXfwF+4zc6+7oh/Ntu6yT8JUvCH7umiDmee649hx8IRDV/ftpu8T7E\nHk49tdWWU/iPPtpK/QTSCp/bOYCu8L1vX3g0b54/BWjjNm0Cjj46PaZbhV+X8Knev9bHFX7Jxiu+\nCU2zgeJ70eIhKfm6Hn7czolRWhz4oSaapZMqjyzl+1dR+ITYn0/ttE0FhC1Lp4cYHU0HCXuJv/3b\n4N2T18pxwgn1CH/vXuCHPwzHEnI4V67y77mnPYefkNt8tX59GEMEBYQPZxXCTyn8p54CVq5sHysR\n/ubNgSiI1LSF4emn2xeQuoSfU/g/+1le4XNLp4rC11R8SuFrHn4J4Wv3khS+tANYU+2SpVMatCVi\np5+nZONViaVT6uHTXDQyl/LwLWi7nzA5GXzzt789kMfixeGP/GUvAz772f23um7bFg4n+cAH5P7j\nj6+32/aWW4IVw5UrYdmyskNMbr0V+KVf6mzPbb564IGwUHAcdFDa0qmi8ONA8vz58lg6pYugKfyn\nnmpPIa1D+N7ng7Zcwcf93rcTfipom7J0YuXdrcIv9eqrePiTk+F3F6dAUuCX3peTe87D975FzvG8\ntY1X2i7XlMLnO2T5+8YLTUmOvVaEzYK2DeKnPwU+/OHwAb/88mB53Hpr+OXv2hXU9he/CFx4Ydq2\naAqrVwMXXwyceKLcX9fS+cY3gIsukvtKT60aHgZ+5Vc623OEf/fdYeHkSCn8yUngkUfag8Mphf/I\nI+02k6bweQ0gQA/arl8fDnch1CF8KgyX2niVsnR27w5tRGJVgrallk4TCr8bDz8mdsq3L7F0pKcB\nIndJ4UuWTh2FH1/LDz/XFH6cgaPl4Wu1e2atwnfOvc4597Bz7qfOuQ/Vvc+OHUG1n3tuUKyjo+EE\nqDvvBN7//kC2VE3xVa8KVsiSJYGIe6n0b7sN+OpXgT//c33MYYeFP7KRkfL7jowA//RPevG1Ektn\n377w+9Eg7k3dAAAgAElEQVQUfqry5a23ht8jR0rhP/JIIFNuaaUUfrw4pBQ+J3wtaFtH4ceETuqe\nAneSgo/TMjmRxoHjbiydOIOnKQ9/bCyMkUolaApf8vBz6ZdSe4mHLyltoN2WSXn4JXn4kn2jtaXy\n8C0tM4Jzbg6AvwPwqwBOA/A259yLSq/fujVsZLr44pClceONwIc+FPzav/7rTsuBY968kDWzdSvw\n8Y93+YMoWL8+1Mz5/Odl24XgXCCj9evL7/2FLwDnn9+eVcJRYunccAPw8pe3PvgcqVLHo6Nhx3Ac\nLE4p/LvuAs46q71NU/hTU63dv3yspvAlSyf+IJUo/I0b0wqfb7oCOgl/165wjXaiVbeEn1L4PA9f\nUuVAmcLnRzxKSj7l4fP754KzQGdaZknOfomlUyXdksZxD5+Tu7QIxItPKgtIWxT4mI0bg5gYBMzr\n8f3PBrDWe/8kADjnvgLgIgAPS4NHRoD/9/+Car71VuDee4FXvxp44xuB//N/Wt5oKYaGgGuvDUT0\nlre0E0y3ePzxQMh/9EfA61+fH0+EzzdQadi5E7jyyrDAaShR+NdeGxYkCalc/B//OKhvUpWElMLn\n9foJmsJfvz78X/K9BSmF//a3t15TmuHERPj/BcKHSiL82I555pn2QHFM+Ny/BzoJPyb0JhX+jh2t\nPQylWToTE2GR5BaSpvCpnTZd0b2o3fsW4e/b125PeN9Z975U4VfdaVti6cTljDVClhaGiYkWSfP3\nkCydks1Yqfx9avvUp8L7X3YZ+o5eWzorADzFXj893daG970v1JxZsQL42MdC22WXBcX1rW8B/+W/\nVCd7wjHHhKeCP/iD5h6vvvtd4Bd/EfjgB0Pd+9J5lCr8T3wCeM1rOj10jpyHv3NnWDDe8ha5P6Xw\nb70V+OVf7mzPKfyY8Enhx7/3hx4Km8nisSUKH+gk6m3bwvVkbUhjgPBkmCL8XJ4+z9CR+ptS+LHC\n1jx8UvdkQWkKP26XPP+9e8Pvg6pX0nuQfy959RLhS8rfezmFU9ppGyt8CrKmqmXmgraxh59T+LHN\nkwva5kov8wWm3+i1wi/CunWrcc45IUB53nmrsCrOQ+wSf/iHIYvmlluA886rf5+JiRAkvuaa4K+/\n4hXl15YS/ubNwN/8TfDeU1i2LNgiGr797eDBawtlSuEPDwOXXNLZTgHTycn2NM/JyZD+GVs6FBjj\nahwA1qwBTjutfSx58963CGxsLPjuXLnTPLiyffJJ4NhjO+/H7aSdO8M1RKo0Jlb4KUtHIvS4n/++\n62bplCp87t/TdVzhU5lqydKJx2tVN8m/L22XVPv4eCu+lhsbp2USqdIO5FzQViJgXrefZ+jw943b\npDz8ycnwjypvSmmZFBSWxnSL4eFhDA8Pd3WPXhP+MwBYKA0rp9vacOONq3s6iaGhkElz2WXBIiJC\nqYLNm4M9MmdOULOcGEpwzDHA976XH3fFFaG0cqxqY+QsnWuvbbdCYtBu2xhjY8CPfgR87Wudfc61\nVD63Y3760/D74GRKoBLJnPAffDA8IXHMmdMiYFJD69cHm4NfC3QStUb4fAzZOfz/Pib0LVvabb8m\nLB26fnw8EJyWOROr+JIsnZjwYyVPQfGY8CVLJ96xy9tzXn2qnat7oCxoG2fuSKof6FT4Wl4838xV\nx8Pnefl0b7548IA2X3CaJvxVq9rF8BVXXFH5Hr22dH4M4CTn3LHOufkAfhPAdT1+TxFvfWv4Y7/h\nhurXPvBACH6ee26wSaqSPRAIKafwH3kkEG2J15eydEZGQkmGCy/Ur9eydO66K+TSx/49QfLx7767\nU90TpJIJksIHOn38OEOHUIfwYztHGpNT+LGlEyv8uJ4PJ1pS8LTgNLHTlpM0XZfL0onTOKXFQ7J0\n4vfWCF/y9nO+PpDeeMXLKMclGHiFy4mJVoyH2ugJQ6pwye+Xastdx4O22gauJgi/CfSU8L33kwB+\nH8BNANYA+Ir3/qFevqeGuXND6uRll1VL07zzzmADXXkl8Bd/0bljtRQlB49//ONh81ZJvCKVpXP9\n9cHO4WQQQ1P4t9/emZ3DIfn4998fYjAS4sAtlWx48Ys7x8Y+frzpihDn4j/5ZHtKJlCP8HMefjdB\nW27nxH1AOi2zrsKXgraxh58jfG7d8PtUydJJBXhjco/TMknhS5uiSH3TuFzgNaXw+Xtolo50nVQf\nX0vvHAT0PA/fe/897/0Lvfcne++v7PX7pfBrvxb+o77+9bLxDz8MvOlNwOc+1/1h5CtWBD86zhwh\nbNkCfPObwH/9r2X3S1k63/wm8J/+U/p6TeHffnt4ktEgKfz77tMJP07NfOqpQCyS/RMr/HjTFSHO\nxd9fCp8XTtP66xC+950Kn/vPk5Ptlkaphy8FbWPFLnn4RFSUuaN5+KUKPz7ohP9scbtk6UgKPybo\nlFWj7aDNXZvL39cUPj/Fq0lLpwn0nPAHCc4BH/0o8D/+h068hI0bgQsuCKr7TW/q/r2HhgKZbNgg\n919zTSDpVD4/h2bpTE0BP/hBmHsKksL3Pk/4msLX0k1jhf/ww+0F2ThihR+fuEWQFH4ThF+SpVM3\naLttW/sixwl/377wt8mJifro0BRuBdVR+NyKkQic20NcDWsevtYu2TdSIJZ+7tRO29RmrFQpBH4/\nCtjytjgtM7cIpOrtxCds5QLC/casInwgpDuuWBFKL2h47rmQW/87vwO8853NvXcqU+c738mrco5l\ny4LCj1Me16wJpEQZGhokhb95c/gDjYmRI1b427cHdXrccfL42MN/4glZtQOdCv/ppzszdIDmFD4n\ndNoJzUm5qqWzfXuniqfr43tL/r7UlyL1UoUfEz4naqmd+ojwqyh8Sc1LCp3GSkRecn1O4XNCTi0M\nJWmZ0qLALR1N4TcdtG0Cs47wnQu5/v/zf8qnRo2PA7/+68A55wCXXtrse2uB2x07QuBTqnmjgXZ8\nxvVwfvhDuZRCDEnhP/BACKamsphihb92baiJM0f5S4otHV7bXhrLiTxF+ETmY2OBTLnVEo8BOjdV\nxWNoNyuP0cSEzw83AToVPvfaqZ8rdb7jmfelrksFZlOLASdqfg1X5tpGKt6XSsuUCF8i4lSWTUkt\nnToKn4/LpWCWqPeSp4BciYZBwKwjfCAcMP62twG/93vtAdyJCeDd7w5/SH/3d/XSN1PQArc33RRI\nmn/gSiDZOv/2b+mgK0FS+GvWpMtVAJ0Kf8OG9rLEMWJLJ0X4XOFPTXXWzOfjiKhJdccLTkz4cQZN\nPIYfXk6ICT8mZt4/Pt6ZrsqJmx9TCLSTJA/YxtfVVfi8THFJdg3fBMX7YusmZw1JB5NrHn6pfVOq\n8Elt87bSFMxcWmYuaKvl+M+qLJ1Bxkc+Ejzb9743qK9164A3vCF88K+9NnyYm4Zm6Vx/fXjvqpAy\ndVIZMxyaws8RfqzwN25M20d1Ff6zzwZFTB9qDm7pxPVv+BhO+LEdE4+pS/gxoXORUEXhx5aOFEzN\n9REhT062WxkpS0dT+FKGTEmWTs7SyXn4WiBXWhg4+ZZm3+SyeaS2lMKnw89L4wP9xqwl/IMOCiUS\n5swJBbVe/vKwc/a669q9zCYhEf7UVNgbUFKPJ0acqTMxESwWLSjKISl8LV2SQ1L4/BzbGLHCj2ve\ncHCFHx9owsGJuoTwx8YC6cRF5PiYOIsGaCf80dHwPZER0G7pxIsB9UtB1LgvvjaVa8/7uIqne3KC\nlXL+U4TPf7Z4I1E8voqlU8XDlyydnMKvmn1Tx8PXbB/Jw09V4uw3BqK0Qr9wyCHAVVeFsguUmdBL\nSIT/7/8elKemelOILZ3HHgvkW2INSQpfy3vnkBT+2Wfr43nQ1vvO1EYOfpqVZMFI40oIn9R7/P/L\nCT2n8ONsGernpB2fdiZdT4jVv9aXsm24GgZai0EcgNUIn34W7wPh8yc1KeskR/hakFSyeWj+uZz7\nEg+/NGhbReFLO22lhWJ8vH3zV7xAGuEPILSAY9OQgrbf/W49OwdoZeoQtB2sEuJaOqOjgYxz2T3x\nubY5hc8tnfiAEGksLQ5x5gpHVYUvkXnJGE7YmoLnhF5X4XNCjPtSHj63Nfh1WgAW0LNxNA9fsiji\ncgmcmGlHa3zv2KbRSJsIlMZqHn5p9k0ctM359dL9+Os4RkC7eDVLJ35K6jdmraXTDxx6aPjD4SRd\n178HOhV+FcJftCgQA6mtZ54JxJ2LXRx8cKelk1okuKUj+egcXLmnCJ8HbUsIX3vfKpaORPglC4IW\ntC0lfJ5tQ32SfQK0yFTy41Ppl9IiIVk6PD9f2lGrzUci/LGx8PujrKhYMfOxUpZPHW9e+5m0a+MT\nr7SALC+UNjFhHr5hGs4Fj3zNmvB606ZQ8bJK1U2OOGhbhfDnzg1/mKTWpfIEEmKFv3FjXuGXEn4V\nhc+tnzrqPR4jET4p1akpnfBTlk5p0DYmfHpf79MKn6tcfs9SS4f6aJHQPPz4KUIj8fjn0DZTxSTO\nx3LCr5qqWbVkAm+L33d0tFWTJ3X/uE4PHZ/I720KfxbjZS8LB7sAIVh7/vmd1SBLEQdtqxA+0F4T\nX9vVGoMr/PHxQJSpYnJULRNoTuFzoo6VszRGuxcfw6tIcpCKTyl477uzdGJlDLSIPefhczLWFH6K\n8FNPBbFFwdslr1pauDRLhpM4HytZOprCT2Xp5FIp47nzEt5z5siLjHR/mi8v6kZPyUT4cTnxfsII\nfz/jZS8LteOBkBH0xjfWvxe3dMbHw9MCPyc2B+7jr1/fuVtVAlf4mzeHAGzqj7kXCp9bOjGREkrJ\nnMbwvPJ4jEb4vHBXTMx0bR2Fz/tjSyf28CWFL2XvcIuoCuHH6lRLgUxZOhph5xQ+f3KIyZcTtPZ0\noAVtpWsnJtqJutT7lyydWOHze/cbRvj7GeecE45w3LUr1Lzppk4Pt3QefTSkMVbZvMUVfqmlwxV+\nLmALtAdtm1T4NC4mUj6GE75UOZQfgxgTIYEIX7sHBW5jwuR9QOeCklP41J9S+LGlQwTJT6Pi16SC\ns1p7bOmk0jVTlo5k/0hzlywdacEYH2+RqKbSpSeBmICltpjwU2mfsaVjCt/QhjPPDCT1+78fyi5r\npFYCbulUtXOA7hV+btMVUC1oW1fh5yydFOHTmFhJE4jwY9Uc96dUuqR+SxW+FmQFyrN06Boaz8mH\nK3zJw5csnclJOV2zxNLRPPyUpVNF4XOyjbN5NDVfovC11M3Y0uEKX1oE+g0j/P2MOXNCXf2bbgL+\n8i+7uxe3dOoQ/iGHBMIE6nn4pQq/6SwdrvBLLR2JzKsSvvT0RLaNpvCJaKnMbtwH6ISvLRS5LB0t\naBs/EfBr6lg6scIvydLRPPyUpaMFS1MKn+fSlyp8TtRk1fB7lVg6scInX39/pX7nMCDTmF1473sD\nWeZ2teawfHnIQPG+HuEvXdqquFlK+HUUPrd0Uoe78MUhLifMEQdtu7F0uiV8sm1SKl3qI8KIM0l4\nP5F0bKnksnS0oK1UxKskLTN+ihgfD32cNKtYOtLPLCn8VOZObN/EKj1lw5Qo/PHx9gUg9VTBd9rG\nHj6dwtXrTZ2lMMKfwVi8OPwBbt0aMn9Kauhw0MatZ58NH0iJ8GIMgsInS0dKWyTsL8InhV+V8Cng\nm7s2p/C1LB0p9VJKD+RplqVZOrTBikgsZeloAV5tcZAsHamthLS19E1pYYiVeWwPSU8B0k7b+D6D\nFLAFjPBnNJwDTjwxZP08/XRZDR2OZctC9kmpugeqK/w6WToUrJS8eaBl6ezeHeYjBcRKCJ9vnNII\nnxR8HAiN7yEtCCl/n+4tkTrvSyn8VJZOjnjjPi2IKgVttYWmRLVrc0mN1VR/TNpaWy4jR3o6kO4l\n2TUpD58If1ACtoAR/ozHSSeFIxtf8pLqSoIUvnSAiAZePK1E4dfJwx8ZCQStfVCIzLUMHT4G6CxA\nFo8hYpVIuQlLJybguL+Owicyk4g3tnpohywpcw4iN27RUDsFZ2OFv3ev7OtrHr5E4twe0camVH9J\naqVk/UhPB1Ib7QSO78XJnBYF+v3mnhQGAUb4MxyveAXwmc+UHXoSgwg/VbI4Bi+eVkXhe6/veOVj\nx8bSdg7QWhi0DB2gReYltg8FOSWftYqlowVtpb64XyN8TeGPjraUpXSNZN3ERE330+6lWTp79+ZT\nNYG0Bx8TYamlk1L4UtC2RM1L4yRLh5M5/73F9+G2D48FDAKM8Gc43vzmQLrveU/1aynoW4XwSeGP\nj4drU7tsgZbCHxkJ16ZqilB55BzhE1FrAVsAP98SL+Wyx/fRcvCBcoUvqfgUofN+LWgrqXUte4df\no3n1e/bIlo6m/LWgrabwOTFTu+bBl46tq/Cr5NxrQdtUJg+VX4hr8kgKf5AIf4AeNgx1cOyx+sHo\nOaxYEbz/8XHg1a8uu4YU/ubNwZ7JPa4S4efsHKCaws8RPtAi9FxapubfA70L2lK/FMDk10p+uZZi\nqT0VUJ9G7Fr2jpaWGd8nRfiapROPTeXS53z4HJFzki5dGDhRa2TOFT5/ouBtgxa0HaCpGPY3XvCC\nQPh79pQrfCLbp5/O+/dAa3NXqr49v3epwidLJ0f4FNxNZeB0S/jk4UtB2xTh8w1RpeRdV+FL3nuu\nvaqlkyLxEg9f2y2rPSHEaaGaKu+Fwo9/b3wM33NgCt8wMHje88LTwc9+Bpx8ctk1zgX749FH8/49\n0CL8JhV+iaVD40ZGAjGmMnlKCV+yfXKWjtZH/amng9HR8PuWgqmx8uf3k85Q1Tz8lPJPZenUtXSq\nevglefipNimzpo6HL90r3pxFY+IsnUFS+Obhz2LQH+LUVLVjHQ8/HLjvvjKFv2RJsFRKCL9U4dNT\nRnxSVIyhobCBS/LvqZ+CtinC13bS8v66Qdt9+wLpxKQwd26Yl6T8JW8f0FU5XVfVw5+Y6CSsKgo/\n5cFLYylFNrZSpOqWOTUvBW2rWD8lefhxJo9070FLyxygtcfQD3zuc9XLMx95JHD33cArX5kf2yuF\nPzpapvBLCF/z+IGyoO3YmK6qcx7+rl3yQe0pgs7ZQFUUforw4w1W2n2a8PDjrBdq6yaXvjRLp24e\nfumYQSJ8U/izHO99L/COd1S75sgjgTvuAI4/Pj924cLwR79pUzolEyhX+AcdFMiohPC3b9cJnzJ5\nRkbShD82JhMs9RNpx2mdJYRPm8ekue3Zoyt8ydJJKfwUscdEy8drTwRNePixpcMtEu2+Oa+/7iIw\nNdWu8Evy8LWAcJyWaZaOYUbjyCMDyeUOPAfCB2Hx4nDAei6Fs1ThU6ZQtwofCB/GHTvSaZlE6HOE\nT8u8eeEJoWoGD9AfhV/F0tHau03LLFX4pXn8UiC4aoB27tzWgl2i3qVMHlP4hgMSK1aEryeeWDZ+\nyZIQ5D366PS4Kgp/z55ywk/VCBoayhP+yIh+zsDcuYHwNULP2UGawu+Vh98NsfPxpR7+5GR+9yy1\nSwo/VUun1OaJ69+k7BvellLvJT7/IAZtB2gqhpmCiy4K9XtKgrZAINxHHwWOOio9jg5L2Z8Knwhf\nW4zmzQvvoxE+KfxcFo6m4nft0q/VFD7l7qeydKrstC0ldm18ytKhapH0dCQRNrVLHn6VWjolJROk\nhYHbN9RWstM2rqipFU8zhW+Y0fiFXwjHM5Zi+fJAEjnCP/TQoKZLFX5Jlk7Kw6cxOYWfInxS+KnS\nCfFhH7w/Z+nEfVS3RVpEuMIv3XiVSstMKf9ShS/ZNLHfTu2xws/V0pE2aOVsmBKFX7rTtvQpwAjf\nMKtwxBHha47wly4NJL5lSzqjhyv8HJk3Yel0q/BThJ8L2kqlKFIBXU3hS0RN7VqQV8vbr+LhS4Qf\n71LVxuaydOqWTK6j8KU8fOmJwsojG2Y9iBwkJctBH+Ddu9PKnWryl6RvNhG0LSH8XHG0lKWj9Uke\nPvWl7B5N4ac8/F5l6cSqndpLCX9ioj2DJpeRk1oEqnr4uZ22kqVjxdMMsx4f+Qjwne+UjaVa71Ll\nSgLV5C8h/P1l6aRKK2spnbmgbZMKv5s8/Lh97968107tWsqnpKolS4cfIUht+0PhS2mZqQyc1JhB\nUvgDNBXDgYpTTgn/SpFTRHz3a2qHcBVLJ5WH/9xz6QWhG0snFbTNKfxULZ0m0jJT5ZTrWjrULmXk\nVLF5cpunNIuljoevZeBIaZkHtIfvnPu4c+4h59w9zrlvOOcOYX2XOufWTve/tvupGmYLSnf+xrtA\npfuUZun0Mmhb19JJKfzdu/Usnar18EuDs9r4OoQvefsli0CuqmbOhpFIeWysnZRjotYCwCVHJR5o\nHv5NAE7z3p8BYC2ASwHAOfdiABcDOBXABQA+7Vzqo2kwBPzu7wKXXFI2NlVbHwhEqp12RRgaSj8p\nNBW0rWrppIK2KYVPp3zFJJNLvyxtp6yeEktH8/BLFX4qN1+zdKqWYJBIOVb9JRk4JbtxBwFdrT3e\n+5vZyx8BeMv09xcC+Ir3fgLAE865tQDOBnBHN+9nOPDxmc+Uj/U+3U82TcrSoQ91L4O2OUtHOl4y\npf5THr5E6tS3c2d5WmbqiaAkX56PjReflG8eX5+qqllSRqGJtMySPHySs5KHn0tW2J9oMmj7HgDX\nT3+/AsBTrO+Z6TaDoTHknhmJ6HMKH8jvtJUIG0j78BS0TVk6uZ22KYVf6tOn+nIKX0vj7DZoW+rX\nxwtDTuH3Ii2Txwi0XbT0txgvAoNm6WSn4pz7PgCeQe0AeAB/5r3/5+kxfwZg3Ht/bU9maTBEeM97\n8nn9TRH+jh36PVKWDp29281OW03hSwFdzbah+2mKfc8ePS0z/r1UIfyUpaMRfipbhtq0ipdaoJWP\nm5wMY/nO31xaZmmdHGmhGLSgbZbwvffnp/qdc+8C8HoAr2bNzwB4AXu9crpNxOrVq3/+/apVq7Bq\n1arctAyzHFddlR9DJE0bvyTkCJ8Urkb4c+eGJwDp+nnzwoc+Req5PHxJrc+fL1+XsnTqePhNWDql\nHj6RO7fGNDLupjImXcfTPLW5VMnD5/fX6u10i+HhYQwPD3d1j66m4px7HYA/BvAq7/0o67oOwN87\n5z6BYOWcBOBO7T6c8A2GprB8eftXCfGhGzGIqFM19Scm9DgBBY4lUp8/P8QHqgZt6TpN4ffa0ikN\n2tLiEMc3JDVPiwDfcEfj+GKay8NPBW1Tap7/H9TZaSu1NR20jcXwFVdcUfke3Xr4fwtgMYDvO+fu\nds59GgC89w8C+CqABxF8/fd7nwuxGQzN4vTTgZNOaj2+S9i9O30PIiuN8Il0c08IEgkPDcnEDbQs\nHWkxGBoKNlLpISfUNzVV7RBzrbRCtx5+bpNV3FZH4WvF03KbrKgt9utLFL4UyJ1xlk4K3vuTE30f\nBfDRbu5vMHSDc88F1q5NjyGFryGn8ImsNYVP1oG06Myfnz8cRXsy2LRJV/hSbINIp2q1zBJLh37G\nEhLX7qG1aYeqSPZNaTmElF8vtWk59jkPfxCDtk1m6RgMMw5TU+n+UsLXFH7quZaILJXDn7KCJA8/\nlbsP6FU0q9TS6YbE+T1ikq2bpZPKyJEOHi9R+Np5takce4nwDyiFbzDMdHzmMyHoqoGIvJeEr1k6\nWt/QUCAziYzHx/XduUB3de9z96lC+HFmUKmlQ0TL58LTJuOTtSYn0wq/VKkDcmaQ9GQwo9MyDYYD\nGWefne6nDJ+6lk6K8Ik0tZTOXJ+k8Hl/SV/VnbZEst0o/NIyCprdktp9K6nrOCOH/1+V2jxSBg5w\nAKZlGgyzGXQO78qVcn9O4aeQUvipktJaH5FxytKRCHxyslz5031KUi1TdlGJ309EXpqbz8dKKZEl\nu2pT7xGTOf990JgDuniawXCg4+STQ+BXeyyn8sy9UviapQPoZFzFw6efq0owl8+BQIq3lPAlO0QL\nBksKP24rtW+ka0sWAVp8YoWf27A1aJaOEb7BkMFJJ+l9ZPUsXSr3j43p16aCtnUUvrYQAHpMIEX4\nqfaYxGlh42p27tzQLhF+fA/N0onHaV4//SyxfVNC7rm0TGnuKUvHFL7BcIDitNOACy7QSzqPjsrt\nQIt8pcJrTRO+ZvfkCF8K5vL3ItDBNSVjqxJ+TuFL5KvthJXuF9tGUhtlc8VptpaWaTDMIhx1FHD9\n9Xq/c7rCS9X6IaKtYulQeypoW0r4Kc+fvxdBInzN79fsH55lw98rJvfJSXmB5XsdKJUyJnLpfiVt\ncfpuytIZZIU/QGuPwXDg4Zpr9NLK1C4FfEvOAY6JhK6RFL5kuQA64eeUf0y40n6GVIA3bpfGaqQa\nt8V92rWl95PGxdAsHX6d9BTQbxjhGww9xNvelh+j7cIFZH9fCwSnLB06GCWGRuy5wLCksGNiq0L4\nqbZSMo4tnXhcNwqfX8+/lwg/FdjtN8zSMRj6hFRRNyL6KuTdDeFX9fYl20gjSC2+EQd447Ep0s4t\nOKVEXuUpQnqCSL2nNKbfMMI3GPqEX/91nYhTdo+W+ZOydKoq/KqWDqAr/Pge5PfzA2xSlk7O5pHe\nv1Thp1R//PNICl+6lyl8g8HQAed0MqDyAZwUCRrhp4K2WpG4qpZOivDjn0WzdKS5lFo6qfev4+GX\nKPXU/U3hGwyGrnHBBcCVV8p9OcLvl8IvJfzSjB6J8GlcTuFrqZrxtaVtvJ3fPxU8HkTCH6CHDYPB\nQFi2DPjQh+S+s84C1q/vbNdIGmiO8KsEbeukcNb11+P37yZAW8XS4WNKUjf7DVP4BsMMw8c+Bjz4\nYGc732Uaoy7h98LSaULhN2XpVMkEylk6MeEPYlqmEb7BMMOQ8v4BOW2zrocft5cGTfnrurtyu7F0\npHG9SMvkY+KfyxS+wWDoKd78ZuD88zvbNYVPG7tiUtIWAinDhhC3aU8D0uJTuvGq1FLqxiLSlHlV\nhW8evsFg6Cn+6Z/k9he+ENixo7M9TiEk5Ai/BE1ZOtKO3H5svMpl8hjhGwyGgcDVV8vtOW8/3gfQ\nBGT+Q8UAAApcSURBVOFLqJqWmapkyb9vKlVTuz8fMxMsnQGaisFg6BWkzB1Ar+ZJJBWTVaq+f4wq\nhF9q6aRUs1Sigj+59Dot0xS+wWAYaLzlLcDmzZ3tGklJpKqBFpm6hJ9KwZRiCNKc+bhuyi3w+fAx\nJVk6g6TwLWhrMMxiHH888Fd/1dmukZRUFVMDpXRqh6VwVCV8aeHJEX6pRVS3tMJMUPhG+AaDoQMX\nXgh89aud7VU8fI3wJWinWwH7X+FrgeGcwtc8fCN8g8Ew0Fi4EPiN35DbS1GF8EuzdOK+XFsdD7+E\n8Es2VQ1i0NYI32AwFOMjHwF+8hO579hj21/T4qAFjDmIFDm50/elCj9H+FUVfkkefjw36f0GSeEP\n0NpjMBgGHYccApxxRmf7mjXA857X3kYKf9my9naJrGlx4IRP13dj6ZSWNC5R+CU+f0nVzX5igKZi\nMBhmKl784s42IuwjjshfLz0N0PdNBW2bVPh1D0npN8zSMRgMPcH8+cArXtF5SPuuXZ1j6YQvOtid\nrgdkwiyp1hmj6k7beFEpIfOqPv/+hil8g8HQEzgH3H57Z/tf/AVw+OHtbUSKvDqnVvYBkAk/Z/OU\n1tLRNpfVVfhm6RgMhlmLD3+4sy1VlE0K+paq5jq1dEoIv0Thm6VjMBgMAlKZPGT3cEgbwHIKv0oZ\nBQlVCT9V6K1fMMI3GAx9x8tfDjz8sNwnEb50zGOO8FNB3aYsHen9TOEbDAYDg3OhhHOM006TM4C0\nom8xJLLlTwdVAqtVK2oOIuEP0MOGwWAwtOOBB+T25cs720o3Y/E26eB3TeGX5NgPuqUzQFMxGAyG\nPJ55ppzw9+zpbOOkTETPCV8rEFdSl5/PYRAVfiOWjnPuj5xzU8655aztUufcWufcQ8651zbxPgaD\nwfD853fW9BkaAs48s3Pszp2dbTnC11B1F+0BqfCdcysBnA/gSdZ2KoCLAZwKYCWAm51zJ3tf5fgE\ng8FgKMPu3Z1K+pRTgHPP7RzLx0mEX2LpaAqfXzuICr+JtecTAP4YwHWs7SIAX/HeTwB4wjm3FsDZ\nAO5o4P0MBoOhDdJGrEcekcdyhU9PCvyJQSP8kkNS+LXUN0iE35Wl45y7EMBT3vv7o64VAJ5ir5+Z\nbjMYDIa+ghd569bSicmc+/8z0tJxzn0fwFG8CYAHcBmADyPYOV1h9erVP/9+1apVWLVqVbe3NBgM\nhg5s3dpevVMi/Be8QL52YqL1vWbp9DItc3h4GMPDw13dw9W11Z1zpwO4GcAehEVgJYKSPxvAewDA\ne3/l9NjvAbjce99h6TjnzNo3GAx9w/vfD3zqUy3VPjoaCrwddlhrjHMhHnDbbeH1T34CnHUWMDIS\nSkbTmKOOAjZtCq9/8APgNa9pH9MknHPw3gu5STpqWzre+we890d770/w3h8P4GkAZ3rvtyD4+W91\nzs13zh0P4CQAd9Z9L4PBYOgVPv3p9nTKBQvayZ7AM340hV9SZrmfaNJd8ghKH977B51zXwXwIIBx\nAO83GW8wGGYqPvlJ4IQTWq+POy58jYPFg15aobal09gEzNIxGAwzELt3A4sWtV47FxaCxx8Pr2+9\nFVi1KgRypU1h3WK/WjoGg8Ewm8HJnsAtHdrl2wuyrwsjfIPBYGgIS5e2vt+9u3/z0DBAGaIGg8Ew\nc7F2bftxjkcf3b+5aDAP32AwGHqEqSm5YmcTMA/fYDAYBgi9Ivu6GLDpGAwGg6FXMMI3GAyGWQIj\nfIPBYJglMMI3GAyGWQIjfIPBYJglMMI3GAyGWQIjfIPBYJglMMI3GAyGWQIjfIPBYJglMMIvRLdH\ni+0v2Dybhc2zOcyEOQIzZ551YIRfiJnyR2DzbBY2z+YwE+YIzJx51oERvsFgMMwSGOEbDAbDLMFA\nlEfu6wQMBoNhhqJqeeS+E77BYDAY9g/M0jEYDIZZAiN8g8FgmCXoK+E7517nnHvYOfdT59yH+jkX\nDc65lc65W5xza5xz9zvnLun3nDQ45+Y45+52zl3X77locM4d6pz7mnPuoenf6Tn9npME59yl0/O7\nzzn39865+f2eEwA4565yzm12zt3H2pY5525yzj3inLvROXdoP+c4PSdpnh+f/n+/xzn3DefcIf2c\n4/ScOubJ+v7IOTflnFvej7lFcxHn6Zz7g+nf6f3OuStz9+kb4Tvn5gD4OwC/CuA0AG9zzr2oX/NJ\nYALAB733pwH4jwD+24DOEwA+AODBfk8ig08CuN57fyqAlwF4qM/z6YBz7lgAvwPgTO/9SwHMA/Cb\n/Z3Vz/EFhM8Mx58CuNl7/0IAtwC4dL/PqhPSPG8CcJr3/gwAazG484RzbiWA8wE8ud9nJKNjns65\nVQDeBOAl3vuXAPjfuZv0U+GfDWCt9/5J7/04gK8AuKiP8xHhvd/kvb9n+vtdCAS1or+z6sT0H+jr\nAXyu33PRMK3ofsl7/wUA8N5PeO+f6/O0JDwHYAzAIufcPAAHA9jQ3ykFeO9vA7A9ar4IwNXT318N\n4M37dVICpHl672/23k9Nv/wRgJX7fWIRlN8nAHwCwB/v5+moUOb5PgBXeu8npsc8m7tPPwl/BYCn\n2OunMYBEyuGcOw7AGQDu6O9MRNAf6CCnXR0P4Fnn3BemrafPOucO6vekYnjvtwP4awDrATwDYIf3\n/ub+ziqJI733m4EgUAAc2ef5lOA9AG7o9yQkOOcuBPCU9/7+fs8lg1MAvMo59yPn3L84516eu8CC\ntoVwzi0G8HUAH5hW+gMD59wbAGyefhJx0/8GEfMAnAXgU977swDsQbAjBgrOuRMA/HcAxwJ4PoDF\nzrm393dWlTDIiz6cc38GYNx7/w/9nkuMaQHyYQCX8+Y+TSeHeQCWee9/EcCfAPhq7oJ+Ev4zAI5h\nr1dOtw0cph/rvw7gS977b/d7PgLOBXChc24dgGsB/Ipz7po+z0nC0wjK6d+nX38dYQEYNLwcwO3e\n+23e+0kA3wTwij7PKYXNzrmjAMA5dzSALX2ejwrn3LsQrMdBXUBPBHAcgHudc48j8NJdzrlBfGp6\nCuFvE977HwOYcs4dlrqgn4T/YwAnOeeOnc6A+E0Ag5pd8nkAD3rvP9nviUjw3n/Ye3+M9/4EhN/j\nLd77d/R7XjGmbYennHOnTDedh8EMMj8C4Bedcwudcw5hnoMUXI6f4q4D8K7p798JYFBESds8nXOv\nQ7AdL/Tej/ZtVp34+Ty99w9474/23p/gvT8eQaSc6b0fhEU0/n//FoBXA8D0Z2rIe781dYO+Ef60\ncvp9hMj9GgBf8d4P0ocKAOCcOxfAbwF4tXPuJ9Pe8+v6Pa8ZjEsA/L1z7h6ELJ3/1ef5dMB7fy+A\nawDcBeBehA/ZZ/s6qWk45/4BwL8BOMU5t945924AVwI43zn3CMLilE3P6zWUef4tgMUAvj/9Ofp0\nXycJdZ4cHgNg6Sjz/DyAE5xz9wP4BwBZkWelFQwGg2GWwIK2BoPBMEtghG8wGAyzBEb4BoPBMEtg\nhG8wGAyzBEb4BoPBMEtghG8wGAyzBEb4BoPBMEtghG8wGAyzBP8fzB70+5WBY1IAAAAASUVORK5C\nYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fe7e6eb36d8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import numpy\n",
    "plot(numpy.arange(0, 15, .01), \n",
    "     [fprime(x) for x in numpy.arange(0, 15, .01)])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Neural Network: XOR"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For this experiment, we will learn the function XOR using back-propagation of error.\n",
    "\n",
    "First, we import the elements from theano and numpy that we will need:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import theano\n",
    "import theano.tensor as T\n",
    "from theano import function, pp\n",
    "import theano.tensor.nnet as nnet\n",
    "import numpy as np\n",
    "import random"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "from matplotlib.pyplot import plot"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Recall XOR:\n",
    "\n",
    "Input 1 | Input 2 | Target\n",
    "--------|---------|-------\n",
    "0 | 0 | 0\n",
    "0 | 1 | 1\n",
    "1 | 0 | 1\n",
    "1 | 1 | 0\n",
    "\n",
    "That is, given 2 inputs the output is True if either of the inputs is True, but not if both are."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Three layers:\n",
    "\n",
    "* 2 inputs (+ 1 bias = 3 values)\n",
    "* 2 hidden units (+ 1 bias = 3 values)\n",
    "* 1 output unit (1 value)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First we define two Theano symbols for representing the inputs and desired output (called the target)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "th_inputs = T.dvector('inputs') # two inputs\n",
    "th_target = T.dscalar('target') # one target/output"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To compute the activation at a layer, we take the inputs * weights to get the net activation, and then apply the sigmoid:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def compute_activation(inputs, weights):\n",
    "    bias = np.array([1], dtype='float64')\n",
    "    all_inputs = T.concatenate([inputs, bias])\n",
    "    net_input = T.dot(weights.T, all_inputs) \n",
    "    activation = nnet.sigmoid(net_input)\n",
    "    return activation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To give you a sense of what compute_activation is, you can see a representation of the computation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'sigmoid((target * join(TensorConstant{0}, inputs, TensorConstant{(1,) of 1.0})))'"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pp(compute_activation(th_inputs, th_target))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What does the sigmoid function return? Let's turn that into a Python function and plot it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "x = T.dscalar('x')          # 64-bit float\n",
    "fx = nnet.sigmoid(x)        # Theano function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "sigmoid.0"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fx"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'sigmoid(x)'"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pp(fx)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "sigmoid = function([x], fx) # Python function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(0.6224593312018546)"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sigmoid(.5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x7fae42bfd2e8>]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAEACAYAAABI5zaHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAGodJREFUeJzt3X90lPWZ9/H3FQRdo9IqGk0oKY2o1YqVVuuv3Q5aFHW7\n+LhsF4HS+FjrWZVud+vWus9JJ2k8Vbd0T59iq0dljd1iqTX+wN2qpMrIamvFI/UXBDCEAAmgPFt/\nEFb5kev5457AMCRkZjLJPXPP53VODjOTe24uQ/LJ5TXf+zvm7oiISLSUhV2AiIjkn8JdRCSCFO4i\nIhGkcBcRiSCFu4hIBCncRUQiaMBwN7MFZrbVzF47yDE/MbO1ZvZHM/tsfksUEZFsZdK53w9c0t8n\nzexSoMbdJwDXAXfnqTYREcnRgOHu7s8DfzrIIdOAnyeP/QMw2swq8lOeiIjkIh8z9ypgY8r9zuRj\nIiISEr2gKiISQYfk4RydwCdS7o9NPnYAM9NGNiIiOXB3y+b4TDt3S370ZTEwB8DMzgHedfetBylQ\nH3n6iMfjodcQpQ99PZ1169Yza1Y9sdj3mDWrnnXr1u/3+Z4eZ8sW57nnnHvvdW66yfmrv3JOPtkx\n+x7gyY/43tsnnfQ9fvELZ/Hi4HkrVjhtbc62bc5HH+3/98+aVQ9sTzmPA9uZNas+4/+GQjhHPmpI\n/chJBid9EOgCPgI2AFcTrIr5RsoxdwJvAa8Ckw5yLpf8icfjYZcQKaX+9Vy3br3X1HzbYbuDO2z3\niopv+7e+td5nz3Y/6yz30aPdjznG/dxz3Wtr3X/wA/fmZvc33nCfMaM+5bnxveeYNat+UDXU1Hzb\n161bX1TnyEcNqZLZmd0vhGyfMJgPhXt+lXoY5Vupfz2nTUsNZ98bSqeeWu/33+/+wgvu27b1//z9\nAy2ec6CtW7feZ82q98mTv+ezZtXnFIiFcI581NArl3A3z7Xlz4GZ+XD+fVGXSCSIxWJhlxEZpfj1\n3LkTHn8c7r0Xli6Ns3t3wwHHTJ4c59lnD3y8L+3tHdTVNfHmm+s47bRP0dhYy/jx1XmuuvSYGZ7l\nzD0fL6hKSEotiIZaKX09V62CBQvg5z+H006Da6+FY44pY9GibqA85chuKiszX1Q3fnw1v/hFPO/1\nSva0FFKkROzYAQ88ABdcABdeCCNHwgsvwNKlMHMm/OAHtdTUxIHu5DO6qamJ09hYG17RkjONZUQi\nonck0tnZQ1VV2d6RyIoVwdhl0SI499ygS7/88iDc+ztHV1cPlZVlGqsUiFzGMgp3kQhob+9gypT5\ntLU1EIxVuhkzJk5FxVw++KCaa66Bq6+GT3xioDNJIVK4i5So2bMbWLjwJtLn5ZMnz6OlJc6IEWFV\nJvmQS7hr5i4SAZ2dPewf7CTv9yjYS5TCXSQCenrK2PdCaK/sVrpItOhfXqTI/fKX8OabtYwdq5Uu\nso9m7iJF7N/+Derq4OmnobxcK12iSi+oipSQO++EH/4QWlrgpJPCrkaGkq5QFSkRd9wB99wDzz0H\nn/xk2NVIIVK4ixQRd4jH4de/hmXLoErveSb9ULiLFAl3uOkmeOaZoGM/7riwK5JCpnAXKQI9PXDD\nDfDKK/Dss3D00WFXJIVO4S5S4Hbvhmuugfb24MXTo44KuyIpBgp3kQK2cyfMng3vvgtPPQWHHx52\nRVIsFO4iBerDD2H6dBgxAhYvhsMOC7siKSa6QlWkAHV3w1/+JRxxBDz8sIJdsqfOXaQApO7Ffuyx\nZbS11TJxYjX33Yc2/pKcKNxFQtbXXuxHHRXnoYfmMmKEtg+Q3GgsIxKyurqmlGAHKOf99xuIx5tC\nrEqKncJdJGT97cXe1dUTRjkSEQp3kZBVVWkvdsk/7QopErL29g4mTpzP9u37Zu41NXFaWuZqy14B\ntCukSFHatauaQw6Zy9/8zTy2bevdi13BLoOjzl0kZHPmwIQJwZtuiPRFb9YhUmTWrIHzz4e33oLR\no8OuRgpVLuGuV2xEQnTrrfD3f69gl/xT5y4SktWr4YIL1LXLwNS5ixQRde0ylNS5i4Sgt2tva9P+\n7DIwde4iReLWW+Fb31Kwy9BR5y4yzNS1S7bUuYsUgcZGde0y9NS5iwyj1lb4i78IVsgo3CVTQ9a5\nm9lUM2s1szVmdnMfnz/KzBab2R/N7HUzq82mCJFSoVm7DJcBO3czKwPWABcBXcByYIa7t6Yccwtw\nlLvfYmZjgNVAhbvvTjuXOncpWeraJVdD1bmfDax19w533wUsAqalHePAkcnbRwL/Lz3YRUqdZu0y\nnDLZFbIK2JhyfxNB4Ke6E1hsZl3AEcDf5qc8kWhobYUlS+Cuu8KuREpFvrb8vQRY4e4XmlkN0GJm\nE919e/qB9fX1e2/HYjFisVieShApXI2N8A//oK5dMpNIJEgkEoM6RyYz93OAenefmrz/XcDd/Y6U\nY/4DuM3dX0jefwa42d1fTjuXZu5Scnpn7W1tcOSRAx8vkm6oZu7LgRPNrNrMRgEzgMVpx3QAX0oW\nUQGcBKzLphCRqOrt2hXsMpwyWuduZlOB/0vwy2CBu99uZtcRdPD3mNkJQBNwQvIpt7n7L/s4jzp3\nKSmrVsEXv6iuXQZHb9YhUmBmzoTTT4dbbgm7EilmCneRAqKuXfJFe8uIFJDGRvjHf1SwSzjUuYsM\nAXXtkk/q3EUKxPe/r65dwqXOXSTPVq6EyZODPWQU7pIP6txFCoBm7VII1LmL5FFv197WBkccEXY1\nEhXq3EVC1tu1K9glbOrcRQapvb2Durom1qzp4bXXynj55Vo+85nqsMuSCNFFTCLDrL29gylT5tPW\n1gCUA93U1MRpaZnL+PEKeMkPjWVEhlldXVNKsAOU09bWQF1dU4hViSjcRQals7OHfcHeq5yurp4w\nyhHZS+EuMghVVWVAd9qj3VRW6kdLwqXvQJFBaGysZdy4OPsCPpi5NzbWhlaTCOgFVZFB++d/7uCh\nh5oYN66HysoyGhtr9WKq5JVWy4iE4LzzoK4OLr007EokqhTuIsOsszN4M44tW2DUqLCrkajSUkiR\nYfboo3D55Qp2KTwKd5FBaG6G6dPDrkLkQBrLiOTonXdgwgTYvBn+7M/CrkaiTGMZkWH02GNwySUK\ndilMCneRHDU3w1//ddhViPRNYxmRHPzpT1BdDV1d2t5Xhp7GMiLD5Ikn4MILFexSuBTuIjnQSEYK\nncYyIln64AOoqoING+BjHwu7GikFGsuIDIP//E84/3wFuxQ2hbtIljSSkWKgsYxIFnbsgBNOgLY2\nGDMm7GqkVGgsIzLEnn4aPv95BbsUPoW7SBY0kpFiobGMSIY++giOPx5WrgxGMyLDRWMZkSH0zDNw\n2mkKdikOCneRDGkkI8VEYxmRDOzeHXTsL78c7CkjMpyGbCxjZlPNrNXM1pjZzf0cEzOzFWb2hpkt\nzaYIkUL33HPwyU8q2KV4HDLQAWZWBtwJXAR0AcvN7HF3b005ZjTwU+Bid+80My0Uk0jRSEaKzYDh\nDpwNrHX3DgAzWwRMA1pTjpkJNLt7J4C7b8t3oSJh2bMHHnkE/uu/wq5EJHOZjGWqgI0p9zclH0t1\nEnC0mS01s+Vm9tV8FSgStt/9DioqgrfUEykWmXTumZ5nEnAhUA783sx+7+5v5en8IqHRSEaKUSbh\n3gmMS7k/NvlYqk3ANnf/EPjQzJYBZwAHhHt9ff3e27FYjFgsll3FIsPIPRjJPPlk2JVIKUkkEiQS\niUGdY8ClkGY2AlhN8ILqZuAl4Cp3X5VyzCnAfGAqcCjwB+Bv3X1l2rm0FFKKyksvwZw5sGoVWFYL\n0UTyJ5elkAN27u6+x8xuBJYQzOgXuPsqM7su+LTf4+6tZvY08BqwB7gnPdhFilHvSEbBLsVGFzGJ\n9MM9eBH1oYdg0qSwq5FSpr1lRPLotdeCZZBnnhl2JSLZU7iL9EMjGSlmCneRfmgJpBQzhbtIH1pb\n4b334AtfCLsSkdwo3EX60NwMV14JZfoJkSKlb12RPjz8sEYyUtwU7iJp1q2Dri644IKwKxHJncJd\nJE1zM1xxBYwYEXYlIrlTuIuk0SoZiQJdoSqSYtMmOOMM2LIFRo4MuxqRgK5QFRmkRx6BL39ZwS7F\nT+EukkIjGYkKjWVEkrZuhZNPDkYyhx0WdjUi+2gsIzIIjz0Gl12mYJdoULiLJGkkI1GSr/dQFSla\n7e0dfOc7TTz7bA9HH13GpEm1jB9fHXZZIoOimbuUtPb2DqZMmU9bWwPBe7t3U1MTp6VlrgJeCoZm\n7iJZqqtrSgl2gHLa2hqoq2sKsSqRwVO4S0nr7OxhX7D3KqerqyeMckTyRuEuJa2qqgzoTnu0m8pK\n/WhIcdPMXUpae3sHp58+n+5uzdylcOUyc1e4S0nr7oaKig6mTm3iv/+7h8rKMhobtVpGCovCXSRL\nzc1w993Q0hJ2JSL902oZkSw1N8P06WFXIZJ/6tylZH30ERx/fPBm2BUVYVcj0j917iJZaGmBiRMV\n7BJNCncpWdpLRqJMYxkpSbt2BSOZV1+FsWPDrkbk4DSWEcnQ0qUwYYKCXaJL4S4lSSMZiTqNZaTk\n7NkDlZXw+9/Dpz4VdjUiA9NYRiQDzz8fhLuCXaJM4S4lRyMZKQUay0hJ6emBceOCNe6f/nTY1Yhk\nRmMZkQG89BKMHq1gl+hTuEtJ0UhGSkVG4W5mU82s1czWmNnNBznuLDPbZWZX5q9Ekfxwh4cfVrhL\naRgw3M2sDLgTuAQ4DbjKzE7p57jbgafzXaRIPqxYASNGBPvJiERdJp372cBad+9w913AImBaH8fN\nBR4G3s5jfSJ50zuSsaxelhIpTpmEexWwMeX+puRje5lZJXCFu98F6EdHCo675u1SWvL1guqPgdRZ\nvAJeCsrKlbBjB5x1VtiViAyPQzI4phMYl3J/bPKxVJ8HFpmZAWOAS81sl7svTj9ZfX393tuxWIxY\nLJZlySLZa26GK6/USEaKQyKRIJFIDOocA17EZGYjgNXARcBm4CXgKndf1c/x9wNPuPsjfXxOFzFJ\nKM44A+68E/78z8OuRCR7uVzENGDn7u57zOxGYAnBGGeBu68ys+uCT/s96U/JpgCRofbWW7B1K5x3\nXtiViAwfbT8gkXfHHbB+Pdx1V9iViORG2w+I9EEXLkkpUucukdbRAZ/7HGzeDCNHhl2NSG7UuYuk\neeQRmDZNwS6lR+EukaYLl6RUaSwjkbV5M5x6KmzZAoceGnY1IrnTWEYkxaOPwuWXK9ilNCncJbI0\nkpFSprGMRNK2bVBTE4xmDj887GpEBkdjGZGkxx+Hiy9WsEvpUrhLJOnCJSl1GstI5Lz7LowbB52d\ncOSRYVcjMngay4gATzwBsZiCXUqbwl0ip7kZpk8PuwqRcGksI5GyfTtUVgZ7ynz842FXI5IfGstI\nyfvNb4J92xXsUuoU7hIpunBJJKCxjETG//wPnHACrF0Lxx4bdjUi+aOxjJS0JUvgzDMV7CKgcJcI\n0UhGZB+NZSQSdu6E44+H11+HqqqwqxHJL41lpGQ98wyccoqCXaSXwl0iQSMZkf0p3KWotbd3MHNm\nAw88EGfZsgba2zvCLkmkIGjmLkWrvb2DKVPm09bWAJQD3dTUxGlpmcv48dVhlyeSN5q5S0mpq2tK\nCXaActraGqirawqxKpHCoHCXotXZ2cO+YO9VTldXTxjliBQUhbsUraqqMqA77dFuKiv1bS2inwIp\nWjfcUEtZWZx9AR/M3Bsba8MrSqRA6AVVKVrf/Cbs2NHBhx820dXVQ2VlGY2NtXoxVSInlxdUFe5S\nlDo74fTTYeXK4MpUkShTuEvJ+OY3YeRI+NGPwq5EZOgp3KUk9Hbtq1ZBRUXY1YgMPYW7lIS5c+HQ\nQ2HevLArERkeCneJPHXtUooU7hJ56tqlFCncJdI2bYKJE9W1S+kZsr1lzGyqmbWa2Rozu7mPz880\ns1eTH8+b2enZFCGSidtvh2uuUbCLZGLAzt3MyoA1wEVAF7AcmOHurSnHnAOscvf3zGwqUO/u5/Rx\nLnXukpPerr21FY47LuxqRIbXUHXuZwNr3b3D3XcBi4BpqQe4+4vu/l7y7ouA3g9H8qq3a1ewi2Tm\nkAyOqQI2ptzfRBD4/fk68ORgihJJtWkTPPhg0LWLSGYyCfeMmdlk4Grggv6Oqa+v33s7FosRi8Xy\nWYJE0G23wde/rq5dSkcikSCRSAzqHJnM3M8hmKFPTd7/LuDufkfacROBZmCqu7f1cy7N3CUrGzfC\nGWdo1i6lbahm7suBE82s2sxGATOAxWl/8TiCYP9qf8Eukovbb1fXLpKLAccy7r7HzG4ElhD8Mljg\n7qvM7Lrg034PUAccDfzMzAzY5e4Hm8uLDGjjRli0KFjXLiLZ0UVMUrCuvx6OPBLuuGPgY0WiTFeo\nSmT0ztpXr4Zjjw27GpFwDdkVqiLD7bbb4NprFewiuVLnLgVn40b47GeDFTIKdxF17hIR6tpFBk+d\nuxSUDRvgzDPVtYukUucuRU9du0h+qHOXgrFhQzBr1woZkf2pc5eidttt8I1vKNhF8kGduxSEjg6Y\nNCno2seMCbsakcKizl2KVm/XrmAXyY+8bvkrko329g7q6ppoa+vhlVfK+N3vaoHqsMsSiQSNZSQU\n7e0dTJkyn7a2BqAc6KamJk5Ly1zGj1fAi6TSWEaKRtCx9wY7QDltbQ3U1TWFWJVIdCjcJRSdnT3s\nC/Ze5XR19YRRjkjkKNwlFKNHlwHdaY92U1mpb0mRfNBPkgy7t96C5ctrOeaYOPsCPpi5NzbWhleY\nSIToBVUZVm++CZdcAvE4fOlLwWqZrq4eKivLaGys1YupIn3Qm3VIQVuxAi67DH74Q5g9O+xqRIpH\nLuGude4yLF58EaZNg7vugiuvDLsakehTuMuQSyTgK1+BpqagcxeRoadwlyH11FMwZw786lcweXLY\n1YiUDq2WkSHz6KPwta/BY48p2EWGm8JdhsSDD8Lf/R08+SScd17Y1YiUHoW75N2CBfBP/wS//W2w\nja+IDD/N3CWvfvIT+NGPghdRJ0wIuxqR0qVwl7y5/Xa47z5YtgyqdS2SSKgU7pKT3r3YOzuDq0uP\nPrqWZ56pZtkyqKwMuzoRUbhL1vrai33UqDgvvDCXykq17CKFQC+oStb62ot9584GfvzjphCrEpFU\nCnfJWmur9mIXKXQKd8nIe+8F+8JMmgStrdqLXaTQ6adR+uUOL7wAtbXB6pelS4MVMa++WktNjfZi\nFylk2vJXDvDOO/Dv/x4sa+zpgWuvha9+FY47bt8xvatltBe7yNDTfu6SsdSljFVVZTQ01LJuXTX3\n3QdPPx1sz3vttXD++WBZfUuJSL4p3CUjfS1lPOSQOBMmzOXGG6uZORM+9rGwqxSRXrmEe0YzdzOb\namatZrbGzG7u55ifmNlaM/ujmX02myIkO+3tHcye3cDkyXFmz26gvb1jwOe4w+bNwbYAM2YcuJRx\n9+4GJk1q4vrrFewiUTDgRUxmVgbcCVwEdAHLzexxd29NOeZSoMbdJ5jZF4C7gXOGqOailj4OyXZW\nvX/XvRw4ixdfjNPSMpfx46v54ANYuxZWr4Y1a/b9uWYNHHYYnHQSbNigpYx9SSQSxGKxsMuIBH0t\nw5dJ5342sNbdO9x9F7AImJZ2zDTg5wDu/gdgtJlV9HWyTDvNdLl0q/k+Rz6eP2XKfBYuvIlEooGF\nC29iypT5GZ9nzx74zndSu+4EUE5bWwPnnNNEVRVUVMDVV8Mjj8CuXTB1Kvz0p9DRAW+/Dc8/Dxdd\npKWMfUkkEmGXEBn6WoYvk+0HqoCNKfc3EQT+wY7pTD62Nf1kCxfetF+nmYm+ZsTDfY6Bnr9nTxCm\nO3cGf6bf3rUrPZihN5inT5/Hl78c5/334f33gzXlff25YweY9d11n3BCD4sXw9ixUDZARjc21vLi\ni/H9/luCpYxzM/paikjhC2FvmSDQzj9/HhMnxvf7TPprrb33X3+9iS1bDgzFc8+dx6mnxnHfd2xf\nt91hzZomtm078Byf+9w8xo2L09PDQT/eeaeJHTsOfP6JJ87DPfjvGDUKRo7c92f67fXr+w7mt9/u\nwT0I5qOOgtGjgz9Tb48eDeXlMGdOGQsXdqedp5vPfKaMceMy+xcYP76alpa51NXNS1nKmPkvShEp\nfAOuljGzc4B6d5+avP9dwN39jpRj7gaWuvuvkvdbgS+6+9a0c2mpjIhIDrJdLZNJ574cONHMqoHN\nwAzgqrRjFgM3AL9K/jJ4Nz3YcylORERyM2C4u/seM7sRWELwAuwCd19lZtcFn/Z73P03ZnaZmb1F\n8Erd1UNbtoiIHMywXsQkIiLDY1jWvpnZdDN7w8z2mNmktM/dkrz4aZWZXTwc9USJmcXNbJOZvZL8\nmBp2TcUmk4v0JHNmtt7MXjWzFWb2Utj1FBszW2BmW83stZTHPm5mS8xstZk9bWajBzrPcC1sfh34\nX8BzqQ+a2aeBrwCfBi4FfmamnUxy8K/uPin58VTYxRSTlIv0LgFOA64ys1PCraro9QAxdz/T3dOX\nTcvA7if4fkz1XeC37n4y8Cxwy0AnGZZwd/fV7r4WSA/uacAid9/t7uuBtRy4hl4Gpl+IucvkIj3J\njqHtxHPm7s8Df0p7eBrwQPL2A8AVA50n7H+A/i5+kuzcmNzT575M/ndN9tPXRXr6HhwcB1rMbLmZ\nXRt2MRFxXO8KRHffAhw3wPH5u4jJzFqA1C0HjOAf+f+4+xP5+ntK0cG+tsDPgO+7u5vZrcC/AtcM\nf5Uie53v7pvN7FiCkF+V7EYlfwZcCZO3cHf3KTk8rRP4RMr9scnHJEUWX9t7Af0izU4nkHptr74H\nB8ndNyf/fMfMHiUYfSncB2ermVW4+1YzOx54e6AnhDGWSZ0PLwZmmNkoMxsPnAjo1fUsJP+he10J\nvBFWLUVq70V6ZjaK4CK9xSHXVLTM7HAzOyJ5uxy4GH1P5sI4MCtrk7e/Bjw+0AmGZW8ZM7sCmA+M\nAf7DzP7o7pe6+0ozewhYCewCrte7eWTtX5L75/cA64Hrwi2nuPR3kV7IZRWzCuDR5FYjhwAL3X1J\nyDUVFTN7EIgBx5jZBiAO3A782sz+N9BBsMrw4OdRloqIRE/Yq2VERGQIKNxFRCJI4S4iEkEKdxGR\nCFK4i4hEkMJdRCSCFO4iIhGkcBcRiaD/D8+xADo3WNVpAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fae43aaa9b0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "xs = range(-10, 10, 1)\n",
    "ys = [sigmoid(x) for x in xs]\n",
    "plot(xs, ys, \"o-\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We create a shared variable named epsilon to control the learning rate. The learning rate will typically be in the range 0.9 to 0.01. This value depends on the function being learned."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "epsilon = theano.shared(0.1, name='epsilon') # learning rate"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we define the method to update the weights. We find the derivative with respect to the weights, and subtract that value from the weights."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def compute_delta_weights(compute_error, weights):\n",
    "    return weights - (epsilon * T.grad(compute_error, wrt=weights))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We define the first set of weights to go between the inputs and the hidden layer. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "NUM_INPUTS = 2\n",
    "NUM_HIDDENS = 2\n",
    "NUM_OUTPUTS = 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The initial random weights are created to span from -1 to 1. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def make_weights(ins, outs):\n",
    "    return np.array(2 * np.random.rand(ins + 1, outs) - 1,\n",
    "                  dtype='float64')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "weights1 = theano.shared(make_weights(NUM_INPUTS, NUM_HIDDENS),\n",
    "                         name='weights1')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "hidden_layer = compute_activation(th_inputs, weights1) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can test the hidden_layer Theano function by turning it into a Python function, and calling it with [0, 0] bound to th_inputs:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0.70860966,  0.59814888])"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "function([th_inputs], hidden_layer)([0, 0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "weights2 = theano.shared(make_weights(NUM_HIDDENS, NUM_OUTPUTS), \n",
    "                         name='weights2') # 4 x 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "output_layer = T.sum(compute_activation(hidden_layer, weights2)) # Theano function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "compute_error = (output_layer - th_target) ** 2  # Theano function"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can test the entire Theano equation now by calling compute_error with values for th_inputs and th_target:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(0.1820956722056817)"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "function([th_inputs, th_target], compute_error)([0, 0], 0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we can create a Python function (called train) that will call compute_error, and update the weights:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "train = function(\n",
    "    inputs=[th_inputs, th_target], \n",
    "    outputs=compute_error, \n",
    "    updates=[(weights1, compute_delta_weights(compute_error, weights1)),\n",
    "             (weights2, compute_delta_weights(compute_error, weights2))])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ok, now we are ready to train a neural network to perform the XOR function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0, 1] 1\n",
      "[1, 0] 1\n",
      "[1, 1] 0\n",
      "[0, 0] 0\n"
     ]
    }
   ],
   "source": [
    "inputs = [[0, 1],\n",
    "          [1, 0],\n",
    "          [1, 1],\n",
    "          [0, 0]]\n",
    "\n",
    "def xor(a, b):\n",
    "    return int((a or b) and not(a and b))\n",
    "\n",
    "for pattern in inputs:\n",
    "    print(pattern, xor(*pattern))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We simply call `train()` on each of the input/target pairs:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def train_all(epochs=5000):\n",
    "    for e in range(epochs):\n",
    "        random.shuffle(inputs)\n",
    "        for i in range(len(inputs)):\n",
    "            target = xor(*inputs[i])\n",
    "            error = train(inputs[i], target) \n",
    "        if (e + 1) % 500 == 0 or e == 0: \n",
    "            print('Epoch:', e + 1, 'error:', error)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 1 error: 0.17590230024029582\n",
      "Epoch: 500 error: 0.21345970272465195\n",
      "Epoch: 1000 error: 0.12049236516936544\n",
      "Epoch: 1500 error: 0.058610122233544594\n",
      "Epoch: 2000 error: 0.021417091129154138\n",
      "Epoch: 2500 error: 0.009440614304426448\n",
      "Epoch: 3000 error: 0.00704809674144815\n",
      "Epoch: 3500 error: 0.0040809552182693035\n",
      "Epoch: 4000 error: 0.0026386841665036777\n",
      "Epoch: 4500 error: 0.0027368789131071805\n",
      "Epoch: 5000 error: 0.002107508288249151\n",
      "CPU times: user 3.96 s, sys: 0 ns, total: 3.96 s\n",
      "Wall time: 3.96 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "train_all()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "test = function([th_inputs], output_layer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(0.04786447075736276)"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test([0, 0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(0.9468603709518669)"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test([0, 1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(0.9541607426531297)"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test([1, 0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(0.04242692545605788)"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test([1, 1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To train another network, we need to reinitialize the weights:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "NUM_HIDDENS = 5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "weights1.set_value(make_weights(NUM_INPUTS, NUM_HIDDENS)) \n",
    "weights2.set_value(make_weights(NUM_HIDDENS, NUM_OUTPUTS)) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We could also try a different learning rate:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "epsilon.set_value(0.05)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 1 error: 0.5129482764713498\n",
      "Epoch: 500 error: 0.27343068436219553\n",
      "Epoch: 1000 error: 0.26608887763525896\n",
      "Epoch: 1500 error: 0.23315802349319936\n",
      "Epoch: 2000 error: 0.27206627125525334\n",
      "Epoch: 2500 error: 0.14440235701111373\n",
      "Epoch: 3000 error: 0.06289470998854373\n",
      "Epoch: 3500 error: 0.032000729577620186\n",
      "Epoch: 4000 error: 0.02204097166832901\n",
      "Epoch: 4500 error: 0.014695800522454913\n",
      "Epoch: 5000 error: 0.010053228953848652\n",
      "CPU times: user 3.96 s, sys: 0 ns, total: 3.96 s\n",
      "Wall time: 3.97 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "train_all()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0, 0] 0.08647435393170991\n",
      "[1, 1] 0.10474032896915239\n",
      "[0, 1] 0.89557766834058\n",
      "[1, 0] 0.9001455909520547\n"
     ]
    }
   ],
   "source": [
    "for pattern in inputs:\n",
    "    print(pattern, test(pattern))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Generalization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Although we only trained on the corners, we can see what "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAQ0AAAEZCAYAAACNVXCFAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAGfpJREFUeJzt3XuwXeVd//H3JxdyDiQEEAm0EQhQcWBUqoWSlpZU2ylF\n22gdKVRbLuo4jg6M8NOG+nPaYdS2KtVW2xlH20rR0lIdTeigQAwpVoFiC4JyaX9pGkqbHKzpryTB\nkNvXP9Y6Jzubvc9Zzzrrts/5vGbOZO+11+W7d/b5nmd91/M8SxGBmVlRC9oOwMxGi5OGmSVx0jCz\nJE4aZpbEScPMkjhpmFkSJw3rDEl3SnpHTfveJen0OvY93zhpdIikqyQ9KmmPpG9J+qik5Qnbb5X0\nYxXGU2h/kk6XdFDSRxL2/R5Jn+xdFhGXRsStZWLt2/e9kq7p2/eyiPj6bPdtThqdIekG4H3ADcCx\nwIXAacA9kha1GVsB7wR2Am+TtLjtYKxmEeGfln+AZcAu4Gf6lh8DPAtclT//BHBTz+sXA9/IH38S\nOAjsAZ4D/g9Z0jkE/BLwzfznhp7tk/Y3Tfz/D/hlYDvw1r7XzgXuBv47f30d8EbghfxnF/Bwvu69\nwDXAUcB3gHN69nMi8Hz+73HAHfln89/545fk6/0OcCBf9zngw/nyQ8AZ+eNj8/f3LLAV+K2e41wJ\n/DPwB2SJcAtwSdvfkS79uKXRDa8ClgB/17swIvYAdwJvmGbbyNd9J/A08JMRcWxE/GHPOmuAM8l+\nWd81wylHkf1NkfQa4KXAp4HPkv3STb62FLgnfw+nAGcB/xQRdwG/B3wmstOGl/e9733A3wJX9Cy+\nDNgcEd8mayF/HPg+4FSyBPGRfNv/S/ZL/2t53Nf2vq/cn5Il6tPzz+adkq7uef0C4Ange8iSx8eG\nf1zzj5NGN5wIfDsiDg14bXv+elEasOy9EbE3Iv6DrHVxxYB1UvbX653AnRHxXeBTwCWSJuP9SWB7\nRPxxROyLiD0R8VDB497WF+fb8/0TETsj4u8i4oU8sb4PeG2R9yFpAfA2YF1EPB8R24Cbgd4C7LaI\n+HhkTY9bgJMlnVQw7jnPSaMbvg2cmH+h+52Sv15WAM/0PN8GvGQW+5siaQz4WQ7/Mj8AfIPsFxyy\nlsCWkru/FxiXdL6k04AfJm+JSRqX9GeSvi7p/wOfB46TNFOCgywBLyJrRU3aRtZamrRj8kFE/A9Z\nwlla8n3MOU4a3XA/2fn9W3sX5s37NwEb80V7gKN7Vjmlbz+DhiyL7Jd30qnAt2axv14/TVYf+Kik\n7ZK2kyWkyVOUb5CdFg0y7b7zVtftZAnoCuBzeasCsmLxy4DzI+I4DrcyJpPGdPv+NrCfrN4z6TSy\neo8V4KTRARHxHHAT8CeS3ihpUd6n4DNkfxH/Kl/1EeBSScdLOhm4rm9XO4AzBhzit/O/zucCV5PV\nH2azv0lXkp3v/yBZS+CHgYuA8/JjfY6saX+tpKMkLZV0Qb7tBHD6DK2D28hOJaZOTXLLgP8BnpN0\nAvDevu0mhsXdk4x+N4/nNODXgVlf6p032q7E+ufwD9kv9GNkLYDtwEeB5T2vLyH7hf8u2S/8dcDT\nPa+/haypvRO4nsNXT36R7C/ptzjy6knS/vpifQmwj54rHD2vfQ74/fzxOWQtpZ358X8zX34CWcFy\nJ/Bv+bJNwDV9+/oq8F/Aop5lp5CdvuwCniS7OnQQWJC/fiHwFNmVlT/Olx3k8NWT48iSxLP5++u/\nenJfXwxT2/onUP6h2ByU/xX9GrA4BhdZzZL59GTuK1IcNCustaQh6RJJT0r6iqR3tRXHPOCmpFWq\nldOT/NLiV4AfJzvPfQi4PCKebDwYM0vSVkvjAuCrEbEtIvaTFePWthSLmSVoayDUS8mu4U96hiyR\nHEGSm9ZmLYmIgfWwro+enFGRToALFhzZoBq0Tf+y/m0GLRu0zsKFCwHYu3cvY2NjU88HrTNp0aIX\n/zf0LyuyzpIlS160zuLFRw46HRsbe9E6Y2NjbN26lVWrVk27Tq/x8fEZ1znmmGOOeH700UfTr3/Z\noHX6jzW5zvr161m7du3AdQbF179s0OfV/x6KfKZF1pn8v/rgBz/I9ddfP+06vfq/K0W+T4PWKfLd\n7V026PWpOIe+Uq9vkvVMnLQS98gza83mzZu57777Cq3bVtJ4CDgr70ewHbictEFUU/oLuYNaEWWK\nvYO2OXSoeFeHiODgwYPJx52N6f46zLTOgQMH2Lt3b9Uh1Wr//v08//zzbYeR5NChQxw4cKDtMF7k\nta99LWvWrJl6ftNNNw1dt5WkEREHJf0a2TwLC4CPRcQTbcRSl0FNxC479thj2w4h2cte9rK2Q0h2\n4YUXth3CrLVW04iIfwTObuv4dRu1pLF8eeFZBTvDSaMd7hFqZkmcNMwsychfci2iqkJokXX6l6UU\nT3sVm0+mO/udKzyAc2ZuaZhZEicNM0vipGFmSeZFTaOMIue2ZesV/QbVGYp0DOvfbtB+9u3bVz6w\nEdK1WkTX4qmSWxpmlsRJw8ySOGmYWRInDTNL0vlCaFOdkaoqXJXp3NVkh6u5WhjtWuGxrg6FXeCW\nhpklcdIwsyROGmaWZKRqGnXVHfqPk7JdFetU1UmsrP6ZpEZxUFud3w07klsaZpbEScPMkjhpmFkS\nJw0zSzJShdCyitzmYKZtZrNdvzpHx5ZZZy5osoDZf6y2i6dNH98tDTNL4qRhZkmcNMwsyZyraZSt\nRZRRZ4eiquoeVc0ANl/V+X/cdi2kLLc0zCyJk4aZJXHSMLMkThpmlqTzhdAFCw7ntaqKg212BKpy\nnf7PwwXM2SvTcatr36e69+2WhpklcdIwsyROGmaWpPM1jd7z9N76xqQy55xVDTxr8liD6jn9n0fb\nM4B1SVX/f1VtM+j/pskZy6ushbilYWZJnDTMLEmtSUPSxyRNSHq0Z9nxku6W9JSkuyQtrzMGM6tW\n3S2NTwBv7Fu2DtgYEWcDm4Aba47BzCpUayE0Ir4g6bS+xWuBi/PHtwCbyRLJQL3FvjpHg5YpYFZ1\nrLKanAGs66qaTavMZ9pkh6v5Wgg9KSImACJiB3BSCzGYWUlduOQ6bQrcv3//1GNJLFy4sPaAzOab\nBx98kC996UuF1m0jaUxIWhERE5JOBp6dbuXFixdPPR7VSUvMuu6Vr3wlr3nNa6aef/jDHx66bhNJ\nQ/nPpA3AVcAHgCuB9dNt3NuyKHK+WWSGpLY7/vTXEEZhBrD+mIvMCDZXNfl9KrPvqjqSDVP3JddP\nAf8KfL+kpyVdDbwfeIOkp4Afz5+b2Yio++rJ24e89Po6j2tm9XGPUDNL4qRhZkm6cMl1WoNGtk6n\nztm96hwdW8V+PQPY7NV1y8U6b2HQ//9X91VGtzTMLImThpklcdIwsySdr2ksWnQ4xEEdiorcUrD/\nnK/tukeZfXsGsNlrskdxmTpD2Y6JTfeUdkvDzJI4aZhZEicNM0vipGFmSTpfCG1q/oyqCk6jeIu+\nuVr4LKOq2zKW2a7J79ds9uOWhpklcdIwsyROGmaWpPM1jd7OXUU6OJXtXNV/jlfnef58ngFsLupa\nHavuDmBuaZhZEicNM0vipGFmSZw0zCxJ5wuhvfc9KVLkLDK1fpGRsIPMhWKgR8KmqXPGtDL/F3Xe\nxqNosdQtDTNL4qRhZkmcNMwsyUjVNLqmqk5QdXYOqmoGsLK1kFFT1YxbZdapajt37jKzTnHSMLMk\nThpmlsRJw8ySjHwhtKqRr2VGxxbpBNVkAayq/Q76LMrEMxcKo1Wp8xYGZfZTdLtB3NIwsyROGmaW\nxEnDzJJ0vqaxZMmSaV+vql5RZKBbEV2fAaxr+y3y+VR1q8siytQQBmmzk1jds3u5pWFmSZw0zCxJ\nrUlD0kpJmyT9p6THJF2bLz9e0t2SnpJ0l6TldcZhZtWpu6VxALg+Is4FVgO/KukHgHXAxog4G9gE\n3FhzHGZWkVoLoRGxA9iRP94t6QlgJbAWuDhf7RZgM1kieZHeQuiggti+ffuqDDlJkzOAFbmFQdHt\nZlqnqs5d/ebTjGB1jU6ts5NY5wqhkk4HzgMeAFZExARMJZaTmorDzGankUuukpYCfwNcl7c4+lPa\n0BT3zDPPTD0eHx9n2bJl9QRpNo9t3br1iN+16dSeNCQtIksYt0bE+nzxhKQVETEh6WTg2WHbr1y5\ncupxm6ciZnPZqlWrOOecc6aeb9y4cei6TbQ0Pg48HhEf6lm2AbgK+ABwJbB+wHYAjI2NTT2uqpNP\nnbd3LKPOGcCqqntUsc0g/e+zv8YxaJ25qqrBjXUPWKs1aUh6NfBzwGOSHiY7DXk3WbK4XdI1wDbg\nsjrjMLPqTJs0JB0LfG9EbOlb/kMR8ehMO4+IfwEWDnn59YWjNLPOGHr1RNJlwJPA3+ads87vefkv\n6w7MzLppukuu7wZ+NCLOA64GbpX00/lrzZ3wm1mnTHd6sjAitgNExBclvQ74nKTvY5pLpFUbHx+f\netxkcbLssZqcAayq4lZdn2tVs3212QGsqiJ1VQYdu8ytGuvq3LVL0pk9O9wOrCHrzXluob2b2Zwz\nXUvjV+g7DYmIXZIuwVc7zOatoUkjIv59yPL9wF/XFpGZdVrnZ+7qrWmMgqpmAKvrvLnNW0BWNQtW\n2Q5gdb33QT2Vi8zcVWaWt7Lfi/7tZvP98iQ8ZpZkxqQh6boiy8xsfijS0rhywLKrKo7DzEbE0JqG\npCuAtwOrJG3oeWkZsLPuwMysm6YrhP4rsB04Ebi5Z/kuYMZxJ1WpohDatVGtZY7dZCejqgqGdc0I\n1rUOYE2qs5Bd1HSXXLeRjUBd3Vw4ZtZ1M15ylbSLw93GjwIWA3si4tg6AzOzbpoxaUTE1Px6ytqX\na4EL6wzKzLorqXNXZCdUfy/pPQyZPbxqxxxzTNL6bdYm6lRnx5+mZiMvMrNY2x3Aiqjq1o1VKTNw\ncTYD1oqcnry15+kC4BXA3kJ7N7M5p0hL4809jw8AXyc7RTGzeahITePqJgIxs9FQ5PTkDOBDZMXP\nAO4Hfj0ivlZzbAAcffTRSet3vU9G08eva1bzJvtyzLTNIHOlL0eRgW9VqfIOa58CbgdOAV4CfBa4\nrXRkZjbSiiSNoyPi1og4kP/8FTA241ZmNicVKYT+g6R1wKfJTk/eBtwp6QSAiPA4FLN5pEjSmJza\n75f7ll9OlkTOqDQiM+u0IldPVjURyDCpnbvaNqhTUZeUHfjWVAewOo/dtVtANvV5QbmZxIYp1CNU\n0quA03vXj4hPFjqCmc0pRS653gqcCTwCTE6AGYCThtk8VKSl8QrgnGi7g72ZdUKRpPEfwMlkE/I0\nLrVz1yjq2t3c2pzcp2zHsjKdz+r8LNr8G1t3zEWSxonA45K+CLzQc4C3lIrMzEZakaTx3rqDMLPR\nUeSS6+ebCMTMRsN0s5F/ISIu6pvuD7L7u4an+zObn6abWPii/N9lw9ZpwnyYuWvQbf2KqOsWkEU6\nPZUpRpbtlFVmP3OlA1hdZlOo7Xb3RTPrnFqThqQlkh6U9LCkx/K5RZF0vKS7JT0l6S5Jy+uMw8yq\nU2vSiIgXgNdFxMuB84A3SbqAbFLijRFxNrAJuLHOOMysOkmzkZcREc/nD5fkxwuyOUYvzpffAmxm\nyOzmozZgrSpl6xxVKNLpqcyAp6o6Zc2VDmBFVDXTeJVqr2lIWiDpYWAHcE9EPASsiIgJgIjYAZxU\ndxxmVo3ak0ZEHMpPT1YCF0g6lyMv4TLguZl1VO2nJ5Mi4jlJm4FLgAlJKyJiQtLJwLPDtrvjjjum\nHp966qmcddZZtcdqNt/s3r2bvXuL3c6o1qQh6URgf0R8V9I48Abg/cAG4CrgA8CVwPph+3jzmw/f\ndmXPnj11hms2by1dupQTTjhh6vmOHTuGrlt3S+MU4BZJC8hOhT4TEXdKegC4XdI1ZHemv2zYDmYa\n5TqKnbnKaLMwCs2NfHUHsPR4yhQ+Z1MsrTVpRMRjwI8MWL4TeH2dxzazerhHqJklcdIwsySNXT0p\naz7M3FVW1zqAVaFIp6wi282nDmBleMCamTXGScPMkjhpmFkSJw0zS9L5Quj4+HjbIYyM+dIBbJAi\nBcw2O4CV7YTY5K0bi3JLw8ySOGmYWRInDTNL0vmaxtjYWNshjLT50gGsX5FaRJMdwAYp8vmU6dhW\ndp2i3NIwsyROGmaWxEnDzJI4aZhZkpEqhHZtlq46p4qva99zsTAKo9kBrF+To2UPHDjwomVFY3ZL\nw8ySOGmYWRInDTNL4ppGB9R9G72ZzMU6R9sdwKpS1cC3fh6wZmaNcdIwsyROGmaWxEnDzJJ0vhC6\nZMmStkOY0mbBssliW9PH79V2B7Ay+6lqJGpZTd8ewS0NM0vipGFmSZw0zCxJ52saRx11VNsh1G4U\nB741eew2bwFZdlasNuscddc43NIwsyROGmaWxEnDzJI4aZhZks4XQhcvXtx2CJVre1TrTOqKb9Bs\nUWU02QGs7KxcVc0kVpUqO/C5pWFmSRpJGpIWSPqypA358+Ml3S3pKUl3SVreRBxmNntNtTSuAx7v\neb4O2BgRZwObgBsbisPMZqn2moaklcClwO8C1+eL1wIX549vATaTJZIXmYs1jSLKnN8WmV276529\n6rzFYRl1ztxV16zmg7YpMqt50VnBmmhp/BHwG0DvO1kRERMAEbEDOKmBOMysArW2NCT9BDAREY9I\nWjPNqkPT6c033zz1+Pzzz2f16tXVBWhmQHZlq2hrre7Tk1cDb5F0KTAOLJN0K7BD0oqImJB0MvDs\nsB3ccMMNU4/3799fc7hm89OiRYuOOD2ZbrLpWk9PIuLdEXFqRJwBXA5sioh3AHcAV+WrXQmsrzMO\nM6tOW5273g/cLukaYBtw2bAVFy2aPsSu39agyQ47Xe801qRR6ABWZJuqiq5FboXQXywdprGkERGf\nBz6fP94JvL6pY5tZddwj1MySOGmYWZLOD1jrrWl0vX4xSNkOTUU6ZTVZw5gL9ZKqBm0VqTO02QGs\nbI2laB3ILQ0zS+KkYWZJnDTMLImThpklmXOF0K4VS7tWQCxbdO3SrRurMug9FRkNWlfnriK68Bm7\npWFmSZw0zCyJk4aZJel8TaPoIJq5pkznrrY7gM2kbF2kyffQH+Og71+ZgYJt1z2qnPl8fv5Gmllp\nThpmlsRJw8ySOGmYWZLOF0J7O3dVdVu/JpUt7HVtBGuZeOqcPasps5nqv1fdxcnU48+mMOuWhpkl\ncdIwsyROGmaWpPM1jd7ONYNmJi8yy3Kbqpqxqcg6bQ9Gq2oG7rr2W+ZYRWbtLlu76drAty7dltHM\n5hAnDTNL4qRhZkmcNMwsSecLoQsXLpx63LUi53wyaiNq61RmJGwRXesANoxbGmaWxEnDzJI4aZhZ\nks7XNFJn7pordY/+88sis2JXNbtXkx2Kun6ssrczrOp72MU6kVsaZpbEScPMkjhpmFkSJw0zS9L5\nQqgkNm/ezJo1a0bqdgb3338/q1evbjuMwrZs2cKZZ54JVDdCtK4C62Thcffu3SxdurSSfdZtMuaD\nBw8e0WGxCnUVeIcZid/CzZs3tx1CsgceeKDtEJJs2bKl7RCS7dmzp+0Qkh08eLDtEGZtJJKGmXWH\nk4aZJVEXO49MktTd4MzmuIgY2EOt00nDzLrHpydmlsRJw8ySdD5pSLpE0pOSviLpXW3HM4ikj0ma\nkPRoz7LjJd0t6SlJd0la3maMvSStlLRJ0n9KekzStfnyTsYsaYmkByU9nMf7nnx5J+PtJWmBpC9L\n2pA/73zMM+l00pC0APhT4I3AucAVkn6g3agG+gRZjL3WARsj4mxgE3Bj41ENdwC4PiLOBVYDv5p/\nrp2MOSJeAF4XES8HzgPeJOkCOhpvn+uAx3uej0LM0+p00gAuAL4aEdsiYj/waWBtyzG9SER8AfhO\n3+K1wC3541uAn2o0qGlExI6IeCR/vBt4AlhJt2N+Pn+4hKwnc9DheCFr0QGXAn/Rs7jTMRfR9aTx\nUuAbPc+fyZeNgpMiYgKyX1LgpJbjGUjS6WR/vR8AVnQ15ryZ/zCwA7gnIh6iw/Hm/gj4DbIEN6nr\nMc+o60ljLunctW1JS4G/Aa7LWxz9MXYm5og4lJ+erAQukHQuHY5X0k8AE3mLbroZeToTc1FdTxrf\nBE7teb4yXzYKJiStAJB0MvBsy/EcQdIisoRxa0Sszxd3OmaAiHgO2AxcQrfjfTXwFklfA24DfkzS\nrcCODsdcSNeTxkPAWZJOk3QUcDmwoeWYhhFH/kXZAFyVP74SWN+/Qcs+DjweER/qWdbJmCWdOHmV\nQdI48AayOkwn4wWIiHdHxKkRcQbZ93ZTRLwDuIOOxlxYRHT6h+wvylPAV4F1bcczJMZPAd8CXgCe\nBq4Gjgc25rHfDRzXdpw98b4aOAg8AjwMfDn/nE/oYszAD+YxPgI8CvxWvryT8Q6I/2JgwyjFPN2P\nu5GbWZKun56YWcc4aZhZEicNM0vipGFmSZw0zCyJk4aZJXHSsIEkfaGGfZ4m6YppXv8HSd+ZHEZu\n3eSkYQNFxEU17HYV8PZpXv994OdrOK5VyEnDBpK0K//3Ykn3SvqspCfy8ROT62yV9AFJj0p6QNIZ\n+fJPSHpr/76A9wEX5ZPSXNd/zIi4F9hd6xuzWXPSsGF6uwqfB1wLnAOcKelVPa99JyJ+CPgI0DuO\nZdC+1gH/HBE/EkeOebER4qRhRXwxIrZHNubgEeD0ntc+nf97G3Bh04FZ85w0rIgXeh4f5Mh7AMeA\nxwfIv1uSBBxVa3TWKCcNG2a6iWN6vS3/93Lg/vzx14FX5I/XAovzx7uAZQWOW/TY1oLO3zXeWjNs\n+HP/8uMl/TuwF5i8nPrnwPp8er67gMk7NT8KHMqX/2V/XUPSfcDZwFJJTwO/EBH3zP6tWJU8NN5K\nk7QV+NGI2Nl2LNYcn57YbPgvzjzkloaZJXFLw8ySOGmYWRInDTNL4qRhZkmcNMwsiZOGmSX5X0cT\nHbApSXE1AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fae3c236630>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "res = 50 # resolution\n",
    "z = np.zeros((res, res))\n",
    "\n",
    "for x in range(res):\n",
    "    for y in range(res):\n",
    "        z[x][y] = test([x/res, y/res])\n",
    "\n",
    "plt.imshow(z, cmap=plt.cm.gray, interpolation='nearest')\n",
    "plt.xlabel(\"input 1\")\n",
    "plt.ylabel(\"input 2\")\n",
    "plt.title(\"Output Activation\")\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Handwriting Categorization\n",
    "\n",
    "First, I'll use the metakernel's %download magic to get the MNIST hand written data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 140,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already up-to-date: metakernel in /home/dblank/.local/lib/python3.4/site-packages\n",
      "Requirement already up-to-date: IPython>=3.0 in /usr/local/lib/python3.4/dist-packages (from metakernel)\n",
      "Requirement already up-to-date: prompt-toolkit<2.0.0,>=1.0.3 in /usr/local/lib/python3.4/dist-packages (from IPython>=3.0->metakernel)\n",
      "Requirement already up-to-date: pygments in /usr/local/lib/python3.4/dist-packages (from IPython>=3.0->metakernel)\n",
      "Requirement already up-to-date: pickleshare in /usr/local/lib/python3.4/dist-packages (from IPython>=3.0->metakernel)\n",
      "Requirement already up-to-date: simplegeneric>0.8 in /usr/lib/python3/dist-packages (from IPython>=3.0->metakernel)\n",
      "Requirement already up-to-date: traitlets>=4.2 in /usr/local/lib/python3.4/dist-packages (from IPython>=3.0->metakernel)\n",
      "Requirement already up-to-date: setuptools>=18.5 in /usr/local/lib/python3.4/dist-packages (from IPython>=3.0->metakernel)\n",
      "Requirement already up-to-date: pexpect; sys_platform != \"win32\" in /usr/local/lib/python3.4/dist-packages (from IPython>=3.0->metakernel)\n",
      "Requirement already up-to-date: decorator in /usr/local/lib/python3.4/dist-packages (from IPython>=3.0->metakernel)\n",
      "Requirement already up-to-date: wcwidth in /usr/local/lib/python3.4/dist-packages (from prompt-toolkit<2.0.0,>=1.0.3->IPython>=3.0->metakernel)\n",
      "Requirement already up-to-date: six>=1.9.0 in /usr/local/lib/python3.4/dist-packages (from prompt-toolkit<2.0.0,>=1.0.3->IPython>=3.0->metakernel)\n",
      "Requirement already up-to-date: ipython-genutils in /usr/local/lib/python3.4/dist-packages (from traitlets>=4.2->IPython>=3.0->metakernel)\n",
      "Requirement already up-to-date: ptyprocess>=0.5 in /usr/local/lib/python3.4/dist-packages (from pexpect; sys_platform != \"win32\"->IPython>=3.0->metakernel)\n"
     ]
    }
   ],
   "source": [
    "! pip install metakernel --user -U"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 141,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import metakernel"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 142,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "metakernel.register_ipython_magics()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloaded 'mnist.pkl.gz'.\n"
     ]
    }
   ],
   "source": [
    "%download http://deeplearning.net/data/mnist/mnist.pkl.gz"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Unzip the file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "!gunzip mnist.pkl.gz"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From http://deeplearning.net/tutorial/gettingstarted.html we see that:\n",
    "\n",
    "> The pickled file represents a tuple of 3 lists: the training set, the validation set and the testing set. Each of the three lists is a pair formed from a list of images and a list of class labels for each of the images. An image is represented as numpy 1-dimensional array of 784 (28 x 28) float values between 0 and 1 (0 stands for black, 1 for white). The labels are numbers between 0 and 9 indicating which digit the image represents."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We read the Python2 pickled data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import pickle\n",
    "import gzip\n",
    "import numpy\n",
    "\n",
    "with open('mnist.pkl', 'rb') as f:\n",
    "    u = pickle._Unpickler(f)\n",
    "    u.encoding = 'latin1'\n",
    "    data = u.load()\n",
    "    train_set, validation_set, test_set = data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(train_set)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "50000"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(train_set[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "784"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(train_set[0][0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "epsilon.set_value(0.1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "weights1.set_value(make_weights(784, 100))\n",
    "weights2.set_value(make_weights(100, 1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "inputs = [train_set[0][i] for i in range(len(train_set[0]))]\n",
    "targets = [train_set[1][i]/9.0 for i in range(len(train_set[0]))]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def display_digit(vector):\n",
    "    for r in range(28):\n",
    "        for c in range(28):\n",
    "            v = int(vector[r * 28 + c] * 10)\n",
    "            ch = \" .23456789\"[v]\n",
    "            print(ch, end=\"\")\n",
    "        print()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                            \n",
      "                            \n",
      "                            \n",
      "                            \n",
      "                            \n",
      "                456.6994    \n",
      "        ..36699999869972    \n",
      "       .99999999993332.     \n",
      "        8999997799          \n",
      "        364998  .6          \n",
      "           693              \n",
      "           597              \n",
      "            792             \n",
      "            .9864           \n",
      "             39994          \n",
      "              .7995.        \n",
      "                3997        \n",
      "                 9992       \n",
      "              .57998        \n",
      "            .5899997        \n",
      "           48999973         \n",
      "         28999973           \n",
      "       68999973             \n",
      "    268999995               \n",
      "    5999855                 \n",
      "                            \n",
      "                            \n",
      "                            \n",
      "5\n",
      "                            \n",
      "                            \n",
      "                            \n",
      "                            \n",
      "               .696.        \n",
      "              .99999        \n",
      "             28999992       \n",
      "            2899973994      \n",
      "           69999993796      \n",
      "          .9997498.396      \n",
      "         .9996 24   99.     \n",
      "        .69983      996     \n",
      "        6992 .      997     \n",
      "       2992         997     \n",
      "       797          997     \n",
      "      2994          995     \n",
      "      398          597      \n",
      "      398         5982      \n",
      "      395       .696        \n",
      "      398      4996         \n",
      "      3995..3689862         \n",
      "      39998899975           \n",
      "      .79999995             \n",
      "        59995.              \n",
      "                            \n",
      "                            \n",
      "                            \n",
      "                            \n",
      "0\n",
      "                            \n",
      "                            \n",
      "                            \n",
      "                            \n",
      "                            \n",
      "                    29.     \n",
      "    23              47.     \n",
      "    46              58.     \n",
      "    86             .96      \n",
      "    86             794      \n",
      "   .96             792      \n",
      "   496             99.      \n",
      "   694            698       \n",
      "   692          36993       \n",
      "   693   ..455999699.       \n",
      "   5998889999753. 99        \n",
      "    46666632     398        \n",
      "                 695        \n",
      "                 692        \n",
      "                 692        \n",
      "                 693        \n",
      "                 693        \n",
      "                 695        \n",
      "                 695        \n",
      "                 395        \n",
      "                            \n",
      "                            \n",
      "                            \n",
      "4\n",
      "                            \n",
      "                            \n",
      "                            \n",
      "                            \n",
      "                            \n",
      "                  4992      \n",
      "                 39992      \n",
      "                 49992      \n",
      "                2998.       \n",
      "               28993        \n",
      "               6997         \n",
      "               9992         \n",
      "             .8994          \n",
      "             4997           \n",
      "            3997            \n",
      "           .9996            \n",
      "           5999.            \n",
      "          .8996             \n",
      "          9997              \n",
      "          9993              \n",
      "         6999.              \n",
      "        .8995               \n",
      "        2998                \n",
      "        2998                \n",
      "         798                \n",
      "                            \n",
      "                            \n",
      "                            \n",
      "1\n",
      "                            \n",
      "                            \n",
      "                            \n",
      "                            \n",
      "                            \n",
      "                            \n",
      "                            \n",
      "            258994352       \n",
      "           3999789996       \n",
      "          29972  7994       \n",
      "         3997   3998        \n",
      "        5995    8993        \n",
      "       4996    3995         \n",
      "       996   .7996          \n",
      "       99. 4799993          \n",
      "       9999998899           \n",
      "       29985. 795           \n",
      "             2994           \n",
      "             299            \n",
      "              99            \n",
      "             299            \n",
      "             499            \n",
      "             .99            \n",
      "              892           \n",
      "              397.          \n",
      "               796          \n",
      "                59.         \n",
      "                            \n",
      "9\n",
      "                            \n",
      "                            \n",
      "                            \n",
      "                            \n",
      "                            \n",
      "                 34         \n",
      "             .589995        \n",
      "           .59998899.       \n",
      "          599998..992       \n",
      "         399998. .992       \n",
      "         79986.  3992       \n",
      "         .52     3992       \n",
      "                 6992       \n",
      "             4444797        \n",
      "           26999998.        \n",
      "         28998699994        \n",
      "        39942  899993       \n",
      "       5997   799569972..   \n",
      "      5994  27982  269998   \n",
      "     4993 26994.     377.   \n",
      "     999889994              \n",
      "     8999964.               \n",
      "      354                   \n",
      "                            \n",
      "                            \n",
      "                            \n",
      "                            \n",
      "                            \n",
      "2\n",
      "                            \n",
      "                            \n",
      "                            \n",
      "                            \n",
      "            598.            \n",
      "           .9992            \n",
      "            6992            \n",
      "            5992            \n",
      "            7992            \n",
      "           .8994.           \n",
      "            49994           \n",
      "             9994           \n",
      "             9994           \n",
      "             9994           \n",
      "             9994           \n",
      "             9994           \n",
      "             9996           \n",
      "             9999.          \n",
      "             59995          \n",
      "             49995          \n",
      "              8999.         \n",
      "              6999.         \n",
      "              2999.         \n",
      "              .899.         \n",
      "                            \n",
      "                            \n",
      "                            \n",
      "                            \n",
      "1\n",
      "                            \n",
      "                            \n",
      "                            \n",
      "                            \n",
      "                            \n",
      "           ..49999996       \n",
      "         .588999999996      \n",
      "         69999999999992     \n",
      "         49985555799992     \n",
      "          ..      8996      \n",
      "                 39995      \n",
      "                37999       \n",
      "              3899997       \n",
      "         .4777999993.       \n",
      "        .8999999996         \n",
      "        .89999999992        \n",
      "         .42....5992        \n",
      "                 992        \n",
      "                3992        \n",
      "       2       39992        \n",
      "     279.     399992        \n",
      "     89955555799973         \n",
      "     899999999985           \n",
      "     .699999885.            \n",
      "       49964                \n",
      "                            \n",
      "                            \n",
      "                            \n",
      "3\n",
      "                            \n",
      "                            \n",
      "                            \n",
      "                            \n",
      "                            \n",
      "             27             \n",
      "             98             \n",
      "             99.            \n",
      "             99.            \n",
      "             992            \n",
      "             994            \n",
      "             995            \n",
      "             695            \n",
      "             695            \n",
      "             696            \n",
      "             699            \n",
      "             499            \n",
      "             399.           \n",
      "             399.           \n",
      "              895           \n",
      "              59            \n",
      "              59            \n",
      "              593           \n",
      "              89.           \n",
      "              68.           \n",
      "                            \n",
      "                            \n",
      "                            \n",
      "1\n",
      "                            \n",
      "                            \n",
      "                            \n",
      "                            \n",
      "                      77    \n",
      "                     595    \n",
      "                    5993    \n",
      "                    797     \n",
      "            27     398.     \n",
      "            892    893      \n",
      "            892   .993      \n",
      "           598    697       \n",
      "          .992   598        \n",
      "         .895   4994        \n",
      "        399974  796         \n",
      "       39998997899          \n",
      "      39992.7999998.        \n",
      "      783   .799996         \n",
      "      4.    2993..          \n",
      "            598             \n",
      "           499.             \n",
      "          .895              \n",
      "          896               \n",
      "          493               \n",
      "                            \n",
      "                            \n",
      "                            \n",
      "                            \n",
      "4\n"
     ]
    }
   ],
   "source": [
    "for i in range(10):\n",
    "    display_digit(inputs[i])\n",
    "    print(int(targets[i] * 9))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def train_digits(max_epochs=1000):\n",
    "    epoch = 0\n",
    "    correct = 0\n",
    "    total = 1\n",
    "    while correct/total < 1.0 and epoch < max_epochs:\n",
    "        total = 0\n",
    "        correct = 0\n",
    "        for i in range(100):\n",
    "            target = targets[i]\n",
    "            error = train(inputs[i], target)\n",
    "            output = test(inputs[i])\n",
    "            target = int(targets[i] * 9)\n",
    "            total += 1\n",
    "            if int(output * 10) == target:\n",
    "                correct += 1\n",
    "        if (epoch + 1) % 10 == 0 or epoch == 0: \n",
    "            print('Epoch:', epoch + 1, 'error:', error, \"% correct:\", correct/total)\n",
    "        epoch += 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 1 error: 0.005584251108050222 % correct: 0.35\n",
      "Epoch: 10 error: 0.0029438254302019415 % correct: 0.63\n",
      "Epoch: 20 error: 0.006812425881126288 % correct: 0.82\n",
      "Epoch: 30 error: 0.007693440417472783 % correct: 0.86\n",
      "Epoch: 40 error: 0.007743773335648142 % correct: 0.87\n",
      "Epoch: 50 error: 0.007499984180644583 % correct: 0.87\n",
      "Epoch: 60 error: 0.006956871273104968 % correct: 0.88\n",
      "Epoch: 70 error: 0.006286300074310477 % correct: 0.91\n",
      "Epoch: 80 error: 0.005599327283432899 % correct: 0.94\n",
      "Epoch: 90 error: 0.004935531392142657 % correct: 0.94\n",
      "Epoch: 100 error: 0.004342748344285977 % correct: 0.94\n",
      "Epoch: 110 error: 0.003883171178634236 % correct: 0.95\n",
      "Epoch: 120 error: 0.0035272259137190546 % correct: 0.96\n",
      "Epoch: 130 error: 0.003215317224888725 % correct: 0.96\n",
      "Epoch: 140 error: 0.0029190018730087764 % correct: 0.96\n",
      "Epoch: 150 error: 0.0026268698087645796 % correct: 0.96\n",
      "Epoch: 160 error: 0.0023347309948372526 % correct: 0.96\n",
      "Epoch: 170 error: 0.002042928231895698 % correct: 0.96\n",
      "Epoch: 180 error: 0.001755288175109719 % correct: 0.96\n",
      "Epoch: 190 error: 0.0014781679125346126 % correct: 0.96\n",
      "Epoch: 200 error: 0.0012190309572252687 % correct: 0.98\n",
      "Epoch: 210 error: 0.0009847410122226002 % correct: 0.98\n",
      "Epoch: 220 error: 0.0007801366103209185 % correct: 0.98\n",
      "Epoch: 230 error: 0.0006073120339109979 % correct: 0.99\n",
      "Epoch: 240 error: 0.0004656801268374049 % correct: 0.99\n",
      "Epoch: 250 error: 0.0003525998252500935 % correct: 0.99\n",
      "Epoch: 260 error: 0.0002642324753321427 % correct: 0.99\n",
      "Epoch: 270 error: 0.00019633693872183434 % correct: 0.99\n"
     ]
    }
   ],
   "source": [
    "train_digits()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "target: 5 output: 0.5556280926515031 correct? True\n",
      "target: 0 output: 0.00696529248487226 correct? True\n",
      "target: 4 output: 0.44446910241180626 correct? True\n",
      "target: 1 output: 0.11612437994607885 correct? True\n",
      "target: 9 output: 0.9812517793933981 correct? True\n",
      "target: 2 output: 0.22215165403428155 correct? True\n",
      "target: 1 output: 0.11598203822154937 correct? True\n",
      "target: 3 output: 0.3346167813701906 correct? True\n",
      "target: 1 output: 0.11068877997250166 correct? True\n",
      "target: 4 output: 0.44540500886593215 correct? True\n",
      "target: 3 output: 0.3341170831349773 correct? True\n",
      "target: 5 output: 0.5568792499273469 correct? True\n",
      "target: 3 output: 0.33388551837641745 correct? True\n",
      "target: 6 output: 0.6672638401988485 correct? True\n",
      "target: 1 output: 0.10927842752617704 correct? True\n",
      "target: 7 output: 0.7789229615036498 correct? True\n",
      "target: 2 output: 0.2225713705572802 correct? True\n",
      "target: 8 output: 0.8908189428421025 correct? True\n",
      "target: 6 output: 0.6677248922319263 correct? True\n",
      "target: 9 output: 0.9793900977855026 correct? True\n",
      "target: 4 output: 0.44469856106545785 correct? True\n",
      "target: 0 output: 0.007001267455873399 correct? True\n",
      "target: 9 output: 0.9846463557436538 correct? True\n",
      "target: 1 output: 0.10996301389789156 correct? True\n",
      "target: 1 output: 0.10971591751434213 correct? True\n",
      "target: 2 output: 0.22123260522741092 correct? True\n",
      "target: 4 output: 0.4437075354301752 correct? True\n",
      "target: 3 output: 0.33245955108642694 correct? True\n",
      "target: 2 output: 0.22139142924010746 correct? True\n",
      "target: 7 output: 0.7777256125206433 correct? True\n",
      "target: 3 output: 0.33281123784607086 correct? True\n",
      "target: 8 output: 0.8862727913564302 correct? True\n",
      "target: 6 output: 0.6655472446852982 correct? True\n",
      "target: 9 output: 0.9854977631290958 correct? True\n",
      "target: 0 output: 0.012859124805210276 correct? True\n",
      "target: 5 output: 0.5541439443323412 correct? True\n",
      "target: 6 output: 0.6658566450892682 correct? True\n",
      "target: 0 output: 0.0045869918515117234 correct? True\n",
      "target: 7 output: 0.7777764954865515 correct? True\n",
      "target: 6 output: 0.665683222751826 correct? True\n",
      "target: 1 output: 0.1118489232630502 correct? True\n",
      "target: 8 output: 0.8902130128625916 correct? True\n",
      "target: 7 output: 0.7787004862466448 correct? True\n",
      "target: 9 output: 0.9770575616968818 correct? True\n",
      "target: 3 output: 0.3329517959962944 correct? True\n",
      "target: 9 output: 0.9948117138012076 correct? True\n",
      "target: 8 output: 0.8871942815407059 correct? True\n",
      "target: 5 output: 0.5543823303022245 correct? True\n",
      "target: 9 output: 0.9769799203817549 correct? True\n",
      "target: 3 output: 0.3300517517507757 correct? True\n",
      "target: 3 output: 0.33032420737682844 correct? True\n",
      "target: 0 output: 0.005608146533500446 correct? True\n",
      "target: 7 output: 0.7749043437453822 correct? True\n",
      "target: 4 output: 0.44243225187476887 correct? True\n",
      "target: 9 output: 0.9931514697143361 correct? True\n",
      "target: 8 output: 0.8901436603592782 correct? True\n",
      "target: 0 output: 0.010464077540657916 correct? True\n",
      "target: 9 output: 0.9807389201659149 correct? True\n",
      "target: 4 output: 0.44191535272546706 correct? True\n",
      "target: 1 output: 0.10258706672261332 correct? True\n",
      "target: 4 output: 0.4400661923621274 correct? True\n",
      "target: 4 output: 0.4403986736570389 correct? True\n",
      "target: 6 output: 0.6624089172163 correct? True\n",
      "target: 0 output: 0.017451950104935568 correct? True\n",
      "target: 4 output: 0.44052409638688433 correct? True\n",
      "target: 5 output: 0.5516453966935414 correct? True\n",
      "target: 6 output: 0.6625686814604919 correct? True\n",
      "target: 1 output: 0.10778971408055982 correct? True\n",
      "target: 0 output: 0.01783609050368972 correct? True\n",
      "target: 0 output: 0.016724211965646964 correct? True\n",
      "target: 1 output: 0.10950419876744502 correct? True\n",
      "target: 7 output: 0.7754287361685435 correct? True\n",
      "target: 1 output: 0.11207989764678129 correct? True\n",
      "target: 6 output: 0.6637885046414723 correct? True\n",
      "target: 3 output: 0.33021325405885893 correct? True\n",
      "target: 0 output: 0.006178701180726108 correct? True\n",
      "target: 2 output: 0.21997331565107614 correct? True\n",
      "target: 1 output: 0.11358695802949283 correct? True\n",
      "target: 1 output: 0.11853774740986184 correct? True\n",
      "target: 7 output: 0.7777038694591021 correct? True\n",
      "target: 9 output: 0.9811237042014678 correct? True\n",
      "target: 0 output: 0.017381357904659216 correct? True\n",
      "target: 2 output: 0.22164797526950986 correct? True\n",
      "target: 6 output: 0.6663031085174334 correct? True\n",
      "target: 7 output: 0.777318729057853 correct? True\n",
      "target: 8 output: 0.8901928915433939 correct? True\n",
      "target: 3 output: 0.33324572168270217 correct? True\n",
      "target: 9 output: 0.9973550323935572 correct? True\n",
      "target: 0 output: 0.014112998658797076 correct? True\n",
      "target: 4 output: 0.44497270900852687 correct? True\n",
      "target: 6 output: 0.6669200576960275 correct? True\n",
      "target: 7 output: 0.7784919375165712 correct? True\n",
      "target: 4 output: 0.44556053293723835 correct? True\n",
      "target: 6 output: 0.6677011153344607 correct? True\n",
      "target: 8 output: 0.8920056061525496 correct? True\n",
      "target: 0 output: 0.008093709424449052 correct? True\n",
      "target: 7 output: 0.7797110840629505 correct? True\n",
      "target: 8 output: 0.8889726548704581 correct? True\n",
      "target: 3 output: 0.3355952176947381 correct? True\n",
      "target: 1 output: 0.10001657749491581 correct? True\n"
     ]
    }
   ],
   "source": [
    "for i in range(100):\n",
    "    output = test(inputs[i])\n",
    "    target = int(targets[i] * 9)\n",
    "    print(\"target:\", target, \"output:\", output, \"correct?\", int(output * 10) == target)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h1>References</h1>\n",
    "\n",
    "1. http://outlace.com/Beginner-Tutorial-Theano/\n",
    "2. http://deeplearning.net/software/theano/tutorial/adding.html\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.1"
  },
  "widgets": {
   "state": {},
   "version": "1.1.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}